[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:40 dist-test-slave-5309 krb5kdc[11434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:40 dist-test-slave-5309 krb5kdc[11434](info): set up 2 sockets
Sep 29 00:49:40 dist-test-slave-5309 krb5kdc[11434](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 00:49:41.791013 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:41 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:41.932979 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33071
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:42.069641 11447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:42.069990 11447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:42.070068 11447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:42.076983 11447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:42.077055 11447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:42.077101 11447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:42.077191 11447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:42.077219 11447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:42.083148 11447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:42.085103 11447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:42.086395 11447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:42.097777 11453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.097927 11455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.097875 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:42.097808 11447 server_base.cc:1034] running on GCE node
I20260929 00:49:42.098834 11447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:42.100296 11447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:42.101500 11447 hybrid_clock.cc:648] HybridClock initialized: now 1790642982101479 us; error 47 us; skew 500 ppm
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:42.104302 11447 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:42.105406 11447 webserver.cc:460] Webserver started at http://127.11.40.254:44391/ using document root <none> and password file <none>
I20260929 00:49:42.106115 11447 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:42.106212 11447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:42.106563 11447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:42.108860 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "35ef59b4a49c4e72b1839291db1661e9"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "bb5c3e10820e1444e37b4ae9b77293a3"
server_key_iv: "5819b1651528b627d3821199834b7668"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.109586 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "35ef59b4a49c4e72b1839291db1661e9"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "bb5c3e10820e1444e37b4ae9b77293a3"
server_key_iv: "5819b1651528b627d3821199834b7668"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.115268 11447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:49:42.118749 11462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.120257 11447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:49:42.120466 11447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "35ef59b4a49c4e72b1839291db1661e9"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "bb5c3e10820e1444e37b4ae9b77293a3"
server_key_iv: "5819b1651528b627d3821199834b7668"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.120615 11447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:42.168448 11447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:42.172226 11447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:42.172502 11447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:42.180524 11447 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33071
I20260929 00:49:42.180536 11517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33071 every 8 connection(s)
I20260929 00:49:42.181751 11447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:42.185420 11518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:42.191622 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 11447
I20260929 00:49:42.191818 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:42.193068 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9: Bootstrap starting.
I20260929 00:49:42.193714 11427 external_mini_cluster.cc:1428] Setting key 9176143aa8243e6ec95160c39d58b989
I20260929 00:49:42.196633 11518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:42.197710 11518 log.cc:824] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:42.200133 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9: No bootstrap required, opened a new log
I20260929 00:49:42.204491 11518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } }
I20260929 00:49:42.204804 11518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:42.204895 11518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ef59b4a49c4e72b1839291db1661e9, State: Initialized, Role: FOLLOWER
I20260929 00:49:42.205394 11518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } }
I20260929 00:49:42.205572 11518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:42.205654 11518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:42.205785 11518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:42.206790 11518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } }
I20260929 00:49:42.207156 11518 leader_election.cc:302] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35ef59b4a49c4e72b1839291db1661e9; no voters:
I20260929 00:49:42.207477 11518 leader_election.cc:288] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:42.207602 11523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:42.207875 11523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [term 1 LEADER]: Becoming Leader. State: Replica: 35ef59b4a49c4e72b1839291db1661e9, State: Running, Role: LEADER
I20260929 00:49:42.208195 11523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } }
I20260929 00:49:42.208976 11518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:42.210186 11525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35ef59b4a49c4e72b1839291db1661e9. Latest consensus state: current_term: 1 leader_uuid: "35ef59b4a49c4e72b1839291db1661e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } } }
I20260929 00:49:42.210323 11525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:42.210763 11532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:42.211635 11524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35ef59b4a49c4e72b1839291db1661e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ef59b4a49c4e72b1839291db1661e9" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33071 } } }
I20260929 00:49:42.211748 11524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:42.214577 11532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:42.221555 11532 catalog_manager.cc:1347] Generated new cluster ID: c49f529e74b34413b769dd63b418572d
I20260929 00:49:42.221627 11532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:42.225849 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:42.197858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.11.40.254:33071)
0929 00:49:42.198423 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:42.198449 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:42.199594 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 00:49:42.204089 (+ 4495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:42.204110 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:42.204528 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:42.204973 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:42.207730 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.209672 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.211880 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.212699 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.212928 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:42.218129 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:42.218184 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:42.218204 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:42.218242 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:42.220704 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.221506 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.221513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.221518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.221696 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.222146 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.222149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.222151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.222690 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:42.222940 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:42.223622 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 00:49:42.224068 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":191,"threads_started":1}
I20260929 00:49:42.279001 11532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:42.280153 11532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:42.284863 11532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35ef59b4a49c4e72b1839291db1661e9: Generated new TSK 0
I20260929 00:49:42.285729 11532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:49:42.325286 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:42.449925 11544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:42.450237 11544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:42.450285 11544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:42.454604 11544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:42.454675 11544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:42.454736 11544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:49:42.460690 11544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:42.461853 11544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:42.462935 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:42.473929 11549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.473948 11550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.473932 11552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:42.474454 11544 server_base.cc:1034] running on GCE node
I20260929 00:49:42.474902 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:42.475641 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:42.476889 11544 hybrid_clock.cc:648] HybridClock initialized: now 1790642982476850 us; error 44 us; skew 500 ppm
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:42.480471 11544 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:49:42.481623 11544 webserver.cc:460] Webserver started at http://127.11.40.193:44273/ using document root <none> and password file <none>
I20260929 00:49:42.482237 11544 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:42.482334 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:42.482582 11544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:42.485045 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "36bde9ce45b4482fb83b8070265602ca"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "95c210700cd58a81dcf6e47ba014b76c"
server_key_iv: "0836e12fd81685602ca0d699127453a2"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.485782 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "36bde9ce45b4482fb83b8070265602ca"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "95c210700cd58a81dcf6e47ba014b76c"
server_key_iv: "0836e12fd81685602ca0d699127453a2"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.490142 11544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:49:42.493086 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.494598 11544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:42.494747 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "36bde9ce45b4482fb83b8070265602ca"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "95c210700cd58a81dcf6e47ba014b76c"
server_key_iv: "0836e12fd81685602ca0d699127453a2"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.494876 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:42.517740 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:42.521662 11544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:42.521920 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:42.522632 11544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:42.523638 11544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:42.523718 11544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.523795 11544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:42.523855 11544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.539100 11544 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:38765
I20260929 00:49:42.539746 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:38765 every 8 connection(s)
I20260929 00:49:42.540385 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:49:42.543661 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 11544
I20260929 00:49:42.543792 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:49:42.544054 11427 external_mini_cluster.cc:1428] Setting key bfe83a5a26ffa0abf6dcce518a3e9d46
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:49:42.556555 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:42.542580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:38537 (local address 127.11.40.254:33071)
0929 00:49:42.542756 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:42.542760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:42.544224 (+ 1464us) server_negotiation.cc:407] Connection header received
0929 00:49:42.545371 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:42.545375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:42.545431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:42.545511 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:42.546783 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.547515 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.548780 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.549276 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.549406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:42.552190 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:42.552196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:42.552199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:42.552231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:42.554060 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.554726 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.554736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.554737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.554817 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.555179 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.555182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.555183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.555427 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:42.555635 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:42.556222 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 00:49:42.556367 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:49:42.557723 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:42.542987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33071 (local address 127.11.40.193:38537)
0929 00:49:42.544006 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:42.544068 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:42.545141 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:42.545710 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:42.545718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:42.546104 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:42.546622 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:42.546636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.547665 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.547668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:42.548648 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:42.548658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.549466 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.549469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:42.549712 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:42.550441 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:42.550458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:42.551985 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:42.554210 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:42.554216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:42.554230 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:42.554611 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:42.554927 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:42.554930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:42.554931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:42.555084 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:42.555636 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:42.555646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:42.555941 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 00:49:42.556177 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:49:42.556556 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":644,"threads_started":1}
I20260929 00:49:42.559437 11676 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33071
I20260929 00:49:42.559727 11676 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:42.560312 11676 heartbeater.cc:511] Master 127.11.40.254:33071 requested a full tablet report, sending...
I20260929 00:49:42.562412 11475 ts_manager.cc:194] Registered new tserver with Master: 36bde9ce45b4482fb83b8070265602ca (127.11.40.193:38765)
I20260929 00:49:42.563632 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:38537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:49:42.577952 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:42.706771 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:42.707056 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:42.707103 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:42.711311 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:42.711378 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:42.711479 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:49:42.717008 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:42.718178 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:42.719262 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:42.730115 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.730032 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:42.730228 11681 server_base.cc:1034] running on GCE node
W20260929 00:49:42.730135 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:42.730772 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:42.731451 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:42.732683 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790642982732666 us; error 28 us; skew 500 ppm
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:42.735548 11681 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:49:42.736435 11681 webserver.cc:460] Webserver started at http://127.11.40.194:44483/ using document root <none> and password file <none>
I20260929 00:49:42.737108 11681 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:42.737203 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:42.737438 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:42.739507 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "acaa8c02bafd4f229af6fdb069f75b23"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "b33a3fa84e1d12e8d3f5b8e68797b9f9"
server_key_iv: "5b64227834d8d81fd42f7224eb513acc"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.740079 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "acaa8c02bafd4f229af6fdb069f75b23"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "b33a3fa84e1d12e8d3f5b8e68797b9f9"
server_key_iv: "5b64227834d8d81fd42f7224eb513acc"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.744493 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:42.747385 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.748899 11681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:42.749037 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "acaa8c02bafd4f229af6fdb069f75b23"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "b33a3fa84e1d12e8d3f5b8e68797b9f9"
server_key_iv: "5b64227834d8d81fd42f7224eb513acc"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.749179 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:42.763320 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:42.766811 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:42.767225 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:42.767935 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:42.768947 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:42.769019 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.769106 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:42.769152 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.780671 11681 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:44179
I20260929 00:49:42.780766 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:44179 every 8 connection(s)
I20260929 00:49:42.781759 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:49:42.785498 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 11681
I20260929 00:49:42.785620 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:49:42.785897 11427 external_mini_cluster.cc:1428] Setting key 99101582643738c2f9df92ccadbd93d3
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:49:42.799088 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:42.783748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:45045 (local address 127.11.40.254:33071)
0929 00:49:42.783931 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:42.783936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:42.785088 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 00:49:42.786714 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:42.786718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:42.786794 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:42.786903 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:42.788534 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.789470 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.791385 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.791751 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.791874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:42.794851 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:42.794860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:42.794862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:42.794888 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:42.796550 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.797231 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.797235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.797237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.797323 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:42.797765 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:42.797768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:42.797769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:42.797969 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:42.798054 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:42.798776 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 00:49:42.798934 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:49:42.800099 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:42.784193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33071 (local address 127.11.40.194:45045)
0929 00:49:42.784896 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:42.784967 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:42.786503 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:42.787100 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:42.787109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:42.787588 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:42.788321 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:42.788349 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.789655 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.789661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:42.791212 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:42.791231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:42.791923 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:42.791928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:42.792119 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:42.792923 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:42.792942 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:42.794657 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:42.796715 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:42.796721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:42.796739 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:42.797108 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:42.797422 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:42.797426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:42.797428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:42.797617 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:42.798069 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:42.798074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:42.798384 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 00:49:42.798654 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 00:49:42.799027 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":218,"threads_started":1}
I20260929 00:49:42.802054 11813 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33071
I20260929 00:49:42.802369 11813 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:42.802935 11813 heartbeater.cc:511] Master 127.11.40.254:33071 requested a full tablet report, sending...
I20260929 00:49:42.804147 11475 ts_manager.cc:194] Registered new tserver with Master: acaa8c02bafd4f229af6fdb069f75b23 (127.11.40.194:44179)
I20260929 00:49:42.805058 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:45045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:49:42.816901 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:42.950260 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:42.950583 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:42.950630 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:42.955111 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:42.955188 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:42.955250 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:49:42.960687 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:42.961889 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:42.962893 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:42.973352 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.973402 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:42.973503 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:42.973402 11818 server_base.cc:1034] running on GCE node
I20260929 00:49:42.974064 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:42.974781 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:42.975950 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790642982975936 us; error 26 us; skew 500 ppm
Sep 29 00:49:42 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:42.978890 11818 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:49:42.979859 11818 webserver.cc:460] Webserver started at http://127.11.40.195:37523/ using document root <none> and password file <none>
I20260929 00:49:42.980480 11818 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:42.980576 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:42.980852 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:42.983012 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "ad50012670e24f81831811db653d6daa"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "24d3148542f516cd2be1b62d7f49919b"
server_key_iv: "f15566e615febff5a444be531bed44c9"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.983614 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "ad50012670e24f81831811db653d6daa"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "24d3148542f516cd2be1b62d7f49919b"
server_key_iv: "f15566e615febff5a444be531bed44c9"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.987967 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:42.990967 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:42.992631 11818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:42.992782 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "ad50012670e24f81831811db653d6daa"
format_stamp: "Formatted at 2026-09-29 00:49:42 on dist-test-slave-5309"
server_key: "24d3148542f516cd2be1b62d7f49919b"
server_key_iv: "f15566e615febff5a444be531bed44c9"
server_key_version: "encryptionkey@0"
I20260929 00:49:42.993053 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:43.010911 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:43.014434 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:43.014606 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:43.015303 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:43.016299 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:43.016347 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:43.016423 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:43.016456 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:43.028184 11818 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:38491
I20260929 00:49:43.028209 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:38491 every 8 connection(s)
I20260929 00:49:43.029192 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:49:43.034842 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 11818
I20260929 00:49:43.034955 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:49:43.035233 11427 external_mini_cluster.cc:1428] Setting key 0ef93eaf68df3ce701cb9c075563bbb1
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:43.044575 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:43.031280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:41907 (local address 127.11.40.254:33071)
0929 00:49:43.031459 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:43.031464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:43.032361 (+ 897us) server_negotiation.cc:407] Connection header received
0929 00:49:43.033759 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:43.033762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:43.033814 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:43.033892 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:43.035190 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.035879 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.037601 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.037937 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.038040 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:43.040850 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:43.040855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:43.040857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:43.040907 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:43.042420 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.042965 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.042970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.042972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.043058 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.043417 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.043420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.043422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.043613 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:43.043721 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:43.044251 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 00:49:43.044388 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:49:43.045531 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:43.031723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33071 (local address 127.11.40.195:41907)
0929 00:49:43.032183 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:43.032227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:43.033539 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:43.034078 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:43.034085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:43.034454 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:43.035033 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:43.035049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.036105 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.036108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:43.037459 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:43.037468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.038125 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.038129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:43.038296 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:43.039030 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:43.039047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:43.040618 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:43.042574 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:43.042580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:43.042594 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:43.042853 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:43.043156 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:43.043160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:43.043162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:43.043310 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:43.043721 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:43.043730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:43.044003 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 00:49:43.044208 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 00:49:43.044553 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260929 00:49:43.046789 11948 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33071
I20260929 00:49:43.047061 11948 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:43.047596 11948 heartbeater.cc:511] Master 127.11.40.254:33071 requested a full tablet report, sending...
I20260929 00:49:43.048754 11475 ts_manager.cc:194] Registered new tserver with Master: ad50012670e24f81831811db653d6daa (127.11.40.195:38491)
I20260929 00:49:43.049446 11475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:41907
I20260929 00:49:43.050792 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:43.080137 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:43.069389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.11.40.193:38765)
0929 00:49:43.069722 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:43.069732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:43.069843 (+ 111us) server_negotiation.cc:407] Connection header received
0929 00:49:43.070070 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:43.070076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:43.070283 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:43.070456 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:43.071213 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.071986 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.072897 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.073287 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.073427 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:43.076209 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:43.076222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:43.076238 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:43.076267 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:43.078145 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.078661 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.078667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.078668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.078764 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.079056 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.079059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.079060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.079380 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:43.079583 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:43.079814 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 00:49:43.079980 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
W20260929 00:49:43.081390 11588 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:43.109457 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:43.100322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.11.40.193:38765)
0929 00:49:43.100541 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:43.100546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:43.100690 (+ 144us) server_negotiation.cc:407] Connection header received
0929 00:49:43.100924 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:43.100932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:43.100978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:43.101073 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:43.101807 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.102320 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.103178 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:43.103421 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:43.103577 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:43.106131 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:43.106135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:43.106137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:43.106161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:43.107652 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.108117 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.108121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.108123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.108209 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:43.108488 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:43.108493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:43.108495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:43.108670 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:43.108852 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:43.109126 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 00:49:43.109274 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 00:49:43.110749 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 11544
I20260929 00:49:43.119359 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 11681
I20260929 00:49:43.126679 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 11818
I20260929 00:49:43.132507 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 11447
2026-09-29T00:49:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2423 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11962](info): set up 2 sockets
Sep 29 00:49:43 dist-test-slave-5309 krb5kdc[11962](info): commencing operation
krb5kdc: starting...
W20260929 00:49:44.193976 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:44 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:44.322091 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44565
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:44565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:44.440726 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:44.441103 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:44.441145 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:44.445254 11975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:44.445322 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:44.445343 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:44.445358 11975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:44.445379 11975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:44.450965 11975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:44565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:44.452127 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:44.453181 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:44.462435 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:44.462486 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:44.462477 11975 server_base.cc:1034] running on GCE node
W20260929 00:49:44.462581 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:44.463270 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:44.464452 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:44.465688 11975 hybrid_clock.cc:648] HybridClock initialized: now 1790642984465624 us; error 77 us; skew 500 ppm
Sep 29 00:49:44 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:44.468742 11975 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:44.469830 11975 webserver.cc:460] Webserver started at http://127.11.40.254:43615/ using document root <none> and password file <none>
I20260929 00:49:44.470453 11975 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:44.470552 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:44.470803 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:44.473116 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "8ad6e66280db4fd7bebe348777ed33b4"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "503dffd32a7332224a1db936a268f7f6"
server_key_iv: "6a74801bca7bf798404c367fae840678"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.473734 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "8ad6e66280db4fd7bebe348777ed33b4"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "503dffd32a7332224a1db936a268f7f6"
server_key_iv: "6a74801bca7bf798404c367fae840678"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.477895 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:49:44.480500 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:44.481861 11975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:49:44.482017 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "8ad6e66280db4fd7bebe348777ed33b4"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "503dffd32a7332224a1db936a268f7f6"
server_key_iv: "6a74801bca7bf798404c367fae840678"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.482141 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:44.514186 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:44.517870 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:44.518204 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:44.525825 11975 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:44565
I20260929 00:49:44.525844 12044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:44565 every 8 connection(s)
I20260929 00:49:44.526830 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:44.529902 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:44.529976 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 11975
I20260929 00:49:44.530077 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:44.530366 11427 external_mini_cluster.cc:1428] Setting key 7a17d5f9005918086037931c8842dddc
I20260929 00:49:44.537091 12045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4: Bootstrap starting.
Sep 29 00:49:44 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:44.540071 12045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:44.541040 12045 log.cc:824] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:44.543329 12045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4: No bootstrap required, opened a new log
I20260929 00:49:44.545609 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:44.531885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.11.40.254:44565)
0929 00:49:44.532376 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:44.532391 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:44.532428 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:49:44.533266 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:44.533288 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:44.533679 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:44.534028 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:44.534850 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.535838 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.536689 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.537257 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.537462 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:44.540494 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:44.540500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:44.540515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:44.540545 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:44.542588 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:44.543077 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:44.543082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:44.543087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:44.543200 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:44.543522 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:44.543526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:44.543528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:44.543939 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:44.544142 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:44.544465 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 00:49:44.544765 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":134,"threads_started":1}
I20260929 00:49:44.547606 12045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } }
I20260929 00:49:44.547932 12045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:44.547991 12045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad6e66280db4fd7bebe348777ed33b4, State: Initialized, Role: FOLLOWER
I20260929 00:49:44.548503 12045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } }
I20260929 00:49:44.548643 12045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:44.548686 12045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:44.548753 12045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:44.549809 12045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } }
I20260929 00:49:44.550338 12045 leader_election.cc:302] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad6e66280db4fd7bebe348777ed33b4; no voters:
I20260929 00:49:44.550786 12045 leader_election.cc:288] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:44.551012 12050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:44.551314 12050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad6e66280db4fd7bebe348777ed33b4, State: Running, Role: LEADER
I20260929 00:49:44.551631 12050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } }
I20260929 00:49:44.552214 12045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:44.553860 12052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad6e66280db4fd7bebe348777ed33b4. Latest consensus state: current_term: 1 leader_uuid: "8ad6e66280db4fd7bebe348777ed33b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } } }
I20260929 00:49:44.554025 12052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:44.554299 12051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad6e66280db4fd7bebe348777ed33b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6e66280db4fd7bebe348777ed33b4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44565 } } }
I20260929 00:49:44.554399 12051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:44.554610 12059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:44.557839 12059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:49:44.565672 12059 catalog_manager.cc:1347] Generated new cluster ID: 61ba618cfb854e85aaeef1e4b999a444
I20260929 00:49:44.565857 12059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:44.583360 12059 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:44.584787 12059 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:44.602792 12059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ad6e66280db4fd7bebe348777ed33b4: Generated new TSK 0
I20260929 00:49:44.603739 12059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:49:44.645555 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:44.767880 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:44.768216 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:44.768268 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:44.773147 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:44.773223 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:44.773334 12071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:49:44.779482 12071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:44.780782 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:44.781975 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:44.793157 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:44.793157 12079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:44.793303 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:44.793541 12071 server_base.cc:1034] running on GCE node
I20260929 00:49:44.794027 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:44.794777 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:44.796002 12071 hybrid_clock.cc:648] HybridClock initialized: now 1790642984795982 us; error 30 us; skew 500 ppm
Sep 29 00:49:44 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:44.799342 12071 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:49:44.800364 12071 webserver.cc:460] Webserver started at http://127.11.40.193:40403/ using document root <none> and password file <none>
I20260929 00:49:44.801047 12071 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:44.801142 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:44.801381 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:44.803912 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "f511cb99fdb049b0926ab60e28dc0199"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "0a3ffdd0461c0a895f56218844751908"
server_key_iv: "f486e40e49268b26011f5471d8142851"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.804549 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "f511cb99fdb049b0926ab60e28dc0199"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "0a3ffdd0461c0a895f56218844751908"
server_key_iv: "f486e40e49268b26011f5471d8142851"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.809464 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:49:44.812342 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:44.813918 12071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:44.814103 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "f511cb99fdb049b0926ab60e28dc0199"
format_stamp: "Formatted at 2026-09-29 00:49:44 on dist-test-slave-5309"
server_key: "0a3ffdd0461c0a895f56218844751908"
server_key_iv: "f486e40e49268b26011f5471d8142851"
server_key_version: "encryptionkey@0"
I20260929 00:49:44.814239 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:44.840338 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:44.844213 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:44.844455 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:44.845223 12071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:44.846146 12071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:44.846195 12071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:44.846237 12071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:44.846253 12071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:44.855969 12071 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:33667
I20260929 00:49:44.856011 12200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:33667 every 8 connection(s)
I20260929 00:49:44.857051 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:49:44.863722 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12071
I20260929 00:49:44.863853 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:49:44.864135 11427 external_mini_cluster.cc:1428] Setting key 2015d7fa6c3620a3757c0ba26e5f3322
Sep 29 00:49:44 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:44.873973 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:44.859065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:40641 (local address 127.11.40.254:44565)
0929 00:49:44.859245 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:44.859250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:44.860131 (+ 881us) server_negotiation.cc:407] Connection header received
0929 00:49:44.861524 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:44.861527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:44.861577 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:44.861653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:44.863078 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.863629 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.865245 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.865606 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.865738 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:44.869324 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:44.869331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:44.869337 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:44.869377 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:44.871336 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:44.871945 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:44.871950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:44.871953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:44.872068 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:44.872497 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:44.872504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:44.872509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:44.872873 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:44.873006 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:44.873578 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 00:49:44.873739 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:49:44.874773 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:44.859483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44565 (local address 127.11.40.193:40641)
0929 00:49:44.859952 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:44.860007 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:44.861293 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:44.861922 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:44.861930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:44.862307 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:44.862881 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:44.862895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.863788 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.863791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:44.865116 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:44.865126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:44.865789 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:44.865800 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:44.866044 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:44.866879 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:44.866905 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:44.869096 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:44.871508 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:44.871514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:44.871529 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:44.871806 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:44.872198 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:44.872201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:44.872202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:44.872363 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:44.872998 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:44.873005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:44.873281 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 00:49:44.873455 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 00:49:44.873740 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:49:44.876247 12201 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44565
I20260929 00:49:44.876520 12201 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:44.877138 12201 heartbeater.cc:511] Master 127.11.40.254:44565 requested a full tablet report, sending...
I20260929 00:49:44.878906 12008 ts_manager.cc:194] Registered new tserver with Master: f511cb99fdb049b0926ab60e28dc0199 (127.11.40.193:33667)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:49:44.880383 12008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:40641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:49:44.897482 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:45.015820 12206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:45.016166 12206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:45.016247 12206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:45.020659 12206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:45.020772 12206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:45.020900 12206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:49:45.026683 12206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:45.028326 12206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:45.029649 12206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:49:45.040545 12206 server_base.cc:1034] running on GCE node
W20260929 00:49:45.040620 12211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:45.040544 12214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:45.040609 12212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:45.041334 12206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:45.042080 12206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:45.043304 12206 hybrid_clock.cc:648] HybridClock initialized: now 1790642985043293 us; error 38 us; skew 500 ppm
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:45.046520 12206 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:49:45.047462 12206 webserver.cc:460] Webserver started at http://127.11.40.194:33273/ using document root <none> and password file <none>
I20260929 00:49:45.048064 12206 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:45.048162 12206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:45.048418 12206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:45.050781 12206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "26ac5947c13c4e05ac6f2897c6c86451"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "bc08480c3c287da46c25b73858719af4"
server_key_iv: "0bda261f41e60d1fa962531721e2cbcf"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.051440 12206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "26ac5947c13c4e05ac6f2897c6c86451"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "bc08480c3c287da46c25b73858719af4"
server_key_iv: "0bda261f41e60d1fa962531721e2cbcf"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.056392 12206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:49:45.059351 12221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:45.061209 12206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:45.061379 12206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "26ac5947c13c4e05ac6f2897c6c86451"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "bc08480c3c287da46c25b73858719af4"
server_key_iv: "0bda261f41e60d1fa962531721e2cbcf"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.061532 12206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:45.082422 12206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:45.086267 12206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:45.086542 12206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:45.087230 12206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:45.088131 12206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:45.088204 12206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:45.088279 12206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:45.088325 12206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:45.099363 12206 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:40585
I20260929 00:49:45.099464 12335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:40585 every 8 connection(s)
I20260929 00:49:45.101130 12206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:49:45.105341 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12206
I20260929 00:49:45.105540 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:49:45.105810 11427 external_mini_cluster.cc:1428] Setting key 962262261602578e460f9d12725bb0de
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:49:45.120110 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.103403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:43511 (local address 127.11.40.254:44565)
0929 00:49:45.103557 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.103562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.104562 (+ 1000us) server_negotiation.cc:407] Connection header received
0929 00:49:45.106266 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.106270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.106349 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.106451 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.108132 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.109028 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.111103 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.111344 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.111445 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.114978 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.114982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.114984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.115021 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.117109 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.117746 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.117749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.117750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.117877 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.118353 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.118358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.118359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.118586 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.118832 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.119724 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.119924 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:49:45.121441 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.103855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44565 (local address 127.11.40.194:43511)
0929 00:49:45.104350 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.104400 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.105990 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.106745 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.106756 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.107336 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:45.107966 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.107983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.109225 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.109228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.110930 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.110945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.111535 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.111542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.111801 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.112628 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:45.112655 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:45.114748 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:45.117261 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:45.117268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:45.117297 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:45.117591 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:45.117980 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:45.117986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:45.117991 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:45.118242 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:45.118832 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:45.118842 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:45.119293 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.119628 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.120053 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260929 00:49:45.122885 12336 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44565
I20260929 00:49:45.123201 12336 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:45.123955 12336 heartbeater.cc:511] Master 127.11.40.254:44565 requested a full tablet report, sending...
I20260929 00:49:45.125649 12008 ts_manager.cc:194] Registered new tserver with Master: 26ac5947c13c4e05ac6f2897c6c86451 (127.11.40.194:40585)
I20260929 00:49:45.127017 12008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:43511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:49:45.145658 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:45.275393 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:45.275720 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:45.275769 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:45.280344 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:45.280447 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:45.280514 12342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:49:45.287567 12342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:45.288862 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:45.290228 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:49:45.304200 12342 server_base.cc:1034] running on GCE node
W20260929 00:49:45.304567 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:45.304194 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:45.304291 12348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:45.305274 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:45.306475 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:45.307747 12342 hybrid_clock.cc:648] HybridClock initialized: now 1790642985307702 us; error 72 us; skew 500 ppm
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:45.311029 12342 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:49:45.312139 12342 webserver.cc:460] Webserver started at http://127.11.40.195:41143/ using document root <none> and password file <none>
I20260929 00:49:45.312783 12342 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:45.312922 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:45.313133 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:45.315402 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "8985c922714f4a33b4607d1c2731e63b"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "b97c6730e77d95174dcd6c4977f8e5cc"
server_key_iv: "9e40cd9a444f0f4edb46b7e3e5b755ed"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.316011 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "8985c922714f4a33b4607d1c2731e63b"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "b97c6730e77d95174dcd6c4977f8e5cc"
server_key_iv: "9e40cd9a444f0f4edb46b7e3e5b755ed"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.320701 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:45.324254 12357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:45.326352 12342 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:49:45.326624 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "8985c922714f4a33b4607d1c2731e63b"
format_stamp: "Formatted at 2026-09-29 00:49:45 on dist-test-slave-5309"
server_key: "b97c6730e77d95174dcd6c4977f8e5cc"
server_key_iv: "9e40cd9a444f0f4edb46b7e3e5b755ed"
server_key_version: "encryptionkey@0"
I20260929 00:49:45.326771 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:45.358870 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:45.362751 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:45.363018 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:45.363806 12342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:45.365099 12342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:45.365180 12342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:45.365365 12342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:45.365543 12342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 00:49:45.381074 12342 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:40133
I20260929 00:49:45.381206 12470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:40133 every 8 connection(s)
I20260929 00:49:45.382157 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:49:45.384404 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12342
I20260929 00:49:45.384531 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:49:45.384903 11427 external_mini_cluster.cc:1428] Setting key 93564d1acd57bf3d67e746635dd2cfe6
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:45.399439 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.384257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:42411 (local address 127.11.40.254:44565)
0929 00:49:45.384493 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.384500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.385864 (+ 1364us) server_negotiation.cc:407] Connection header received
0929 00:49:45.387362 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.387374 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.387443 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.387667 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.389033 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.389691 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.390984 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.391235 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.391367 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.395099 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.395103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.395105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.395138 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.396939 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.397586 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.397595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.397597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.397689 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.398174 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.398177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.398179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.398406 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.398523 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.399116 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.399263 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 00:49:45.400729 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.384776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44565 (local address 127.11.40.195:42411)
0929 00:49:45.385504 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.385577 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.387094 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.387901 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.387909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.388299 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:45.388879 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.388906 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.389874 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.389877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.390837 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.390855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.391401 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.391405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.391609 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.392462 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:45.392481 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:45.394844 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:45.397105 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:45.397112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:45.397129 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:45.397445 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:45.397806 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:45.397813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:45.397815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:45.398051 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:45.398523 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:45.398532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:45.398837 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.399058 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.399450 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":151,"threads_started":1}
I20260929 00:49:45.402196 12471 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44565
I20260929 00:49:45.402501 12471 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:45.403071 12471 heartbeater.cc:511] Master 127.11.40.254:44565 requested a full tablet report, sending...
I20260929 00:49:45.404418 12008 ts_manager.cc:194] Registered new tserver with Master: 8985c922714f4a33b4607d1c2731e63b (127.11.40.195:40133)
I20260929 00:49:45.405412 12008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:42411
I20260929 00:49:45.412366 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:49:45.426252 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.417312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.11.40.254:44565)
0929 00:49:45.417565 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.417571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.417740 (+ 169us) server_negotiation.cc:407] Connection header received
0929 00:49:45.417966 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.417970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.418018 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.418096 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.419024 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.419616 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.420522 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.420757 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.420979 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.422745 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.422749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.422751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.422781 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.424290 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.424870 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.424874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.424876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.424976 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.425289 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.425292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.425293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.425564 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.425690 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.425933 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.426045 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260929 00:49:45.433979 12008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:49:45.437659 12008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:45.455569 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.448481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33667 (local address 127.0.0.1:60824)
0929 00:49:45.449379 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.449420 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.449603 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.450312 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.450320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.450357 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:45.450522 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.450531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.451878 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.451882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.453366 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.453377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.454888 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.454891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.455077 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.455125 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.455251 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.455409 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260929 00:49:45.455613 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.448662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.11.40.193:33667)
0929 00:49:45.449110 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.449118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.449786 (+ 668us) server_negotiation.cc:407] Connection header received
0929 00:49:45.449910 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.449917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.450149 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.450325 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:45.450643 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.451747 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.453515 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.454751 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.454886 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.455120 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.455299 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.455409 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260929 00:49:45.455947 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.448749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40585 (local address 127.0.0.1:49094)
0929 00:49:45.449659 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.449674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.449809 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.451282 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.451285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.451304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:45.451488 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.451498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.453132 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.453136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.454268 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.454279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.455531 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.455534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.455661 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.455685 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.455737 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.455807 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":245,"threads_started":1}
I20260929 00:49:45.457078 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.449852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.11.40.194:40585)
0929 00:49:45.450765 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.450774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.450819 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:49:45.450906 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.450912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.451130 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.451293 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:45.451632 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.452863 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.454515 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.455391 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.456485 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.456694 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.456755 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.456907 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":128,"threads_started":1}
I20260929 00:49:45.457759 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.449376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:40133 (local address 127.0.0.1:44220)
0929 00:49:45.450437 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.450451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.450547 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.452066 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.452068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.452085 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:45.452206 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.452212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.454101 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.454104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.455359 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.455371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.457200 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.457204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.457356 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.457386 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.457525 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.457613 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":75,"thread_start_us":103,"threads_started":1}
I20260929 00:49:45.458261 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.449570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.11.40.195:40133)
0929 00:49:45.450139 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.450181 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.451483 (+ 1302us) server_negotiation.cc:407] Connection header received
0929 00:49:45.451571 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.451578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.451745 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.451909 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:45.452338 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.453822 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.455527 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.456881 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.456991 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.457133 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.457946 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.458061 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":295,"threads_started":1}
I20260929 00:49:45.459842 12135 tablet_service.cc:1467] Processing CreateTablet for tablet a2f791098c224dafb07d90ed353fb62c (DEFAULT_TABLE table=test-table [id=a7310914053342299dbc84b117da4ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:45.461017 12135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f791098c224dafb07d90ed353fb62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:45.460736 12270 tablet_service.cc:1467] Processing CreateTablet for tablet a2f791098c224dafb07d90ed353fb62c (DEFAULT_TABLE table=test-table [id=a7310914053342299dbc84b117da4ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:45.461050 12405 tablet_service.cc:1467] Processing CreateTablet for tablet a2f791098c224dafb07d90ed353fb62c (DEFAULT_TABLE table=test-table [id=a7310914053342299dbc84b117da4ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:45.462040 12270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f791098c224dafb07d90ed353fb62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:45.462090 12405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f791098c224dafb07d90ed353fb62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:45.470392 12490 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Bootstrap starting.
I20260929 00:49:45.470738 12488 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Bootstrap starting.
I20260929 00:49:45.473294 12489 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Bootstrap starting.
I20260929 00:49:45.473320 12488 tablet_bootstrap.cc:654] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:45.473616 12490 tablet_bootstrap.cc:654] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:45.474360 12488 log.cc:824] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:45.474362 12490 log.cc:824] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:45.475725 12489 tablet_bootstrap.cc:654] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:45.477061 12488 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: No bootstrap required, opened a new log
I20260929 00:49:45.477310 12488 ts_tablet_manager.cc:1397] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 00:49:45.477344 12489 log.cc:824] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:45.477440 12490 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: No bootstrap required, opened a new log
I20260929 00:49:45.477631 12490 ts_tablet_manager.cc:1397] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260929 00:49:45.479774 12489 tablet_bootstrap.cc:492] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: No bootstrap required, opened a new log
I20260929 00:49:45.479976 12489 ts_tablet_manager.cc:1397] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:49:45.481374 12488 raft_consensus.cc:348] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.481643 12488 raft_consensus.cc:374] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:45.481716 12488 raft_consensus.cc:729] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f511cb99fdb049b0926ab60e28dc0199, State: Initialized, Role: FOLLOWER
I20260929 00:49:45.481669 12490 raft_consensus.cc:348] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.481964 12490 raft_consensus.cc:374] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:45.482070 12490 raft_consensus.cc:729] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ac5947c13c4e05ac6f2897c6c86451, State: Initialized, Role: FOLLOWER
I20260929 00:49:45.482301 12488 consensus_queue.cc:260] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.482517 12490 consensus_queue.cc:260] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.483091 12201 heartbeater.cc:503] Master 127.11.40.254:44565 was elected leader, sending a full tablet report...
I20260929 00:49:45.483523 12488 ts_tablet_manager.cc:1428] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:49:45.483868 12490 ts_tablet_manager.cc:1428] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:49:45.484326 12336 heartbeater.cc:503] Master 127.11.40.254:44565 was elected leader, sending a full tablet report...
I20260929 00:49:45.484872 12489 raft_consensus.cc:348] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.485193 12489 raft_consensus.cc:374] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:45.485251 12489 raft_consensus.cc:729] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8985c922714f4a33b4607d1c2731e63b, State: Initialized, Role: FOLLOWER
I20260929 00:49:45.485746 12489 consensus_queue.cc:260] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.486651 12471 heartbeater.cc:503] Master 127.11.40.254:44565 was elected leader, sending a full tablet report...
I20260929 00:49:45.487676 12489 ts_tablet_manager.cc:1428] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260929 00:49:45.602775 12337 tablet.cc:2369] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:49:45.609243 12202 tablet.cc:2369] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:49:45.633251 12472 tablet.cc:2369] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:49:45.651419 12495 raft_consensus.cc:482] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:45.651758 12495 raft_consensus.cc:504] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.653285 12495 leader_election.cc:288] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f511cb99fdb049b0926ab60e28dc0199 (127.11.40.193:33667), 8985c922714f4a33b4607d1c2731e63b (127.11.40.195:40133)
I20260929 00:49:45.657403 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.653738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:34709 (local address 127.11.40.195:40133)
0929 00:49:45.653970 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.653976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.653999 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:49:45.654193 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.654196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.654246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.654336 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:45.654622 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.655139 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.656325 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.656738 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.656967 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.657022 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.657200 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.657273 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 00:49:45.657429 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.653582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:40133 (local address 127.11.40.194:34709)
0929 00:49:45.653915 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.653931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.654058 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.654349 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.654352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.654368 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:45.654506 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.654512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.655295 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.655298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.656201 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.656210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.656961 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.656964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.657062 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.657094 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.657185 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.657261 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260929 00:49:45.657608 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.653739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:54029 (local address 127.11.40.193:33667)
0929 00:49:45.653924 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.653929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.654085 (+ 156us) server_negotiation.cc:407] Connection header received
0929 00:49:45.654289 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.654298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.654348 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.654434 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:45.654716 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.655292 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.656371 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.657028 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.657163 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.657214 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.657409 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.657480 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 00:49:45.657759 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.653588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33667 (local address 127.11.40.194:54029)
0929 00:49:45.653961 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:45.653974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:45.654157 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:45.654449 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:45.654451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:45.654468 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:45.654603 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.654608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.655409 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.655421 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.656243 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:45.656252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.657154 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.657157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:45.657272 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.657301 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:49:45.657394 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:49:45.657472 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":83,"threads_started":1}
I20260929 00:49:45.658411 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f791098c224dafb07d90ed353fb62c" candidate_uuid: "26ac5947c13c4e05ac6f2897c6c86451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8985c922714f4a33b4607d1c2731e63b" is_pre_election: true
I20260929 00:49:45.658581 12155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f791098c224dafb07d90ed353fb62c" candidate_uuid: "26ac5947c13c4e05ac6f2897c6c86451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f511cb99fdb049b0926ab60e28dc0199" is_pre_election: true
I20260929 00:49:45.658815 12425 raft_consensus.cc:2454] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ac5947c13c4e05ac6f2897c6c86451 in term 0.
I20260929 00:49:45.658821 12155 raft_consensus.cc:2454] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ac5947c13c4e05ac6f2897c6c86451 in term 0.
I20260929 00:49:45.659415 12222 leader_election.cc:302] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26ac5947c13c4e05ac6f2897c6c86451, 8985c922714f4a33b4607d1c2731e63b, f511cb99fdb049b0926ab60e28dc0199; no voters:
I20260929 00:49:45.659754 12495 raft_consensus.cc:2781] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:45.659845 12495 raft_consensus.cc:482] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:45.659929 12495 raft_consensus.cc:3037] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:45.661391 12495 raft_consensus.cc:504] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.661749 12495 leader_election.cc:288] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [CANDIDATE]: Term 1 election: Requested vote from peers f511cb99fdb049b0926ab60e28dc0199 (127.11.40.193:33667), 8985c922714f4a33b4607d1c2731e63b (127.11.40.195:40133)
I20260929 00:49:45.662158 12155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f791098c224dafb07d90ed353fb62c" candidate_uuid: "26ac5947c13c4e05ac6f2897c6c86451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f511cb99fdb049b0926ab60e28dc0199"
I20260929 00:49:45.662201 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f791098c224dafb07d90ed353fb62c" candidate_uuid: "26ac5947c13c4e05ac6f2897c6c86451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8985c922714f4a33b4607d1c2731e63b"
I20260929 00:49:45.662276 12155 raft_consensus.cc:3037] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:45.662456 12425 raft_consensus.cc:3037] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:45.663878 12155 raft_consensus.cc:2454] T a2f791098c224dafb07d90ed353fb62c P f511cb99fdb049b0926ab60e28dc0199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ac5947c13c4e05ac6f2897c6c86451 in term 1.
I20260929 00:49:45.663905 12425 raft_consensus.cc:2454] T a2f791098c224dafb07d90ed353fb62c P 8985c922714f4a33b4607d1c2731e63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ac5947c13c4e05ac6f2897c6c86451 in term 1.
I20260929 00:49:45.664278 12225 leader_election.cc:302] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ac5947c13c4e05ac6f2897c6c86451, f511cb99fdb049b0926ab60e28dc0199; no voters:
I20260929 00:49:45.664533 12495 raft_consensus.cc:2781] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:45.664920 12495 raft_consensus.cc:686] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [term 1 LEADER]: Becoming Leader. State: Replica: 26ac5947c13c4e05ac6f2897c6c86451, State: Running, Role: LEADER
I20260929 00:49:45.665318 12495 consensus_queue.cc:237] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } }
I20260929 00:49:45.668776 12005 catalog_manager.cc:5495] T a2f791098c224dafb07d90ed353fb62c P 26ac5947c13c4e05ac6f2897c6c86451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ac5947c13c4e05ac6f2897c6c86451 (127.11.40.194). New cstate: current_term: 1 leader_uuid: "26ac5947c13c4e05ac6f2897c6c86451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511cb99fdb049b0926ab60e28dc0199" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ac5947c13c4e05ac6f2897c6c86451" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8985c922714f4a33b4607d1c2731e63b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40133 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:45.706959 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.695523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.11.40.193:33667)
0929 00:49:45.695692 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.695696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.695918 (+ 222us) server_negotiation.cc:407] Connection header received
0929 00:49:45.696118 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.696122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.696169 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.696266 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.697162 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.697688 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.698857 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.699241 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.699378 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.702347 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.702362 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.702378 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.702409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.704334 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.705148 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.705155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.705156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.705351 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.705746 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.705749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.705750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.706066 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.706238 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.706563 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.706760 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:45.739686 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.729418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.11.40.193:33667)
0929 00:49:45.729694 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.729697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.729855 (+ 158us) server_negotiation.cc:407] Connection header received
0929 00:49:45.730211 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.730215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.730308 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.730403 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.731265 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.731944 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.733005 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.733243 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.733340 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.736328 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.736332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.736334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.736366 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.738059 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.738539 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.738542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.738543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.738622 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.738881 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.738883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.738885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.739067 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.739169 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.739444 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.739554 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[11962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:45.769949 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:45.759712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.11.40.193:33667)
0929 00:49:45.759990 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:45.759994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:45.760011 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:49:45.760072 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:45.760074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:45.760116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:45.760185 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:45.761131 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.762143 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.763030 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:45.763242 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:45.763327 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:45.766130 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:45.766133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:45.766135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:45.766165 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:45.767785 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.768277 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.768280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.768281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.768405 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:45.768700 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:45.768703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:45.768704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:45.768984 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:45.769113 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:45.769638 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 00:49:45.769817 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
W20260929 00:49:45.771579 12115 tablet_service.cc:2908] Rejecting scan request for tablet a2f791098c224dafb07d90ed353fb62c: Uninitialized: clean time has not yet been initialized
I20260929 00:49:45.774012 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12071
I20260929 00:49:45.780665 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12206
I20260929 00:49:45.787043 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12342
I20260929 00:49:45.793919 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 11975
2026-09-29T00:49:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2664 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[12513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[12513](info): set up 2 sockets
Sep 29 00:49:45 dist-test-slave-5309 krb5kdc[12513](info): commencing operation
krb5kdc: starting...
W20260929 00:49:46.882134 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:46 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:47.013684 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33345
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:47.128090 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:47.128448 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:47.128530 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:47.132689 12526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:47.132803 12526 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:49:47.132851 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:47.132899 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:47.132956 12526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:47.132998 12526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:47.139797 12526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33345
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:47.141065 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:47.142159 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:47.151757 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.151889 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.151862 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:47.151780 12526 server_base.cc:1034] running on GCE node
I20260929 00:49:47.152617 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:47.153860 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:47.155059 12526 hybrid_clock.cc:648] HybridClock initialized: now 1790642987155047 us; error 27 us; skew 500 ppm
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:47.158139 12526 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:47.159209 12526 webserver.cc:460] Webserver started at http://127.11.40.254:37511/ using document root <none> and password file <none>
I20260929 00:49:47.159907 12526 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:47.160004 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:47.160247 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:47.162402 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "54865ce54f904f5fae86bead2f84d3db"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "fa4f639bd9a83300f00b124ade1fa697"
server_key_iv: "832e6e405e804e3e045b9bef6b914458"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.163012 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "54865ce54f904f5fae86bead2f84d3db"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "fa4f639bd9a83300f00b124ade1fa697"
server_key_iv: "832e6e405e804e3e045b9bef6b914458"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.166873 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:49:47.169415 12541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.170698 12526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:49:47.170845 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "54865ce54f904f5fae86bead2f84d3db"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "fa4f639bd9a83300f00b124ade1fa697"
server_key_iv: "832e6e405e804e3e045b9bef6b914458"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.170961 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:47.190073 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:47.193782 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:47.194015 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:47.203246 12526 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33345
I20260929 00:49:47.203246 12603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33345 every 8 connection(s)
I20260929 00:49:47.204289 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:47.207307 12604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:47.210940 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12526
I20260929 00:49:47.211071 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:47.211324 11427 external_mini_cluster.cc:1428] Setting key d06549b1f382192ada213860f4358cbd
I20260929 00:49:47.213757 12604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Bootstrap starting.
I20260929 00:49:47.216074 12604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:47.216879 12604 log.cc:824] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:47.218983 12604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: No bootstrap required, opened a new log
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:47.222970 12604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } }
I20260929 00:49:47.223222 12604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:47.223285 12604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54865ce54f904f5fae86bead2f84d3db, State: Initialized, Role: FOLLOWER
I20260929 00:49:47.223786 12604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } }
I20260929 00:49:47.223963 12604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:47.224051 12604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:47.224179 12604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:47.225241 12604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } }
I20260929 00:49:47.225622 12604 leader_election.cc:302] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54865ce54f904f5fae86bead2f84d3db; no voters:
I20260929 00:49:47.226047 12604 leader_election.cc:288] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:47.226131 12609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:47.226382 12609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [term 1 LEADER]: Becoming Leader. State: Replica: 54865ce54f904f5fae86bead2f84d3db, State: Running, Role: LEADER
I20260929 00:49:47.226707 12609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } }
I20260929 00:49:47.227174 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.212968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.11.40.254:33345)
0929 00:49:47.213448 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:47.213463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:47.213501 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:49:47.214268 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:47.214288 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:47.214666 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:47.215021 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:47.215850 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.217016 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.218119 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.218607 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.218799 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.221635 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:47.221640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:47.221656 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:47.221685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:47.223670 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.224216 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.224221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.224226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.224398 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.224723 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.224726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.224728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.225232 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:47.225467 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:47.225772 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 00:49:47.226118 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
I20260929 00:49:47.227501 12604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:47.227967 12609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54865ce54f904f5fae86bead2f84d3db. Latest consensus state: current_term: 1 leader_uuid: "54865ce54f904f5fae86bead2f84d3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } } }
I20260929 00:49:47.228085 12609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:47.228600 12610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54865ce54f904f5fae86bead2f84d3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54865ce54f904f5fae86bead2f84d3db" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33345 } } }
I20260929 00:49:47.228794 12610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:47.229547 12613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:47.232694 12613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:49:47.238322 12613 catalog_manager.cc:1347] Generated new cluster ID: bd846e830d24458593c272df3e32bcb5
I20260929 00:49:47.238408 12613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:47.268198 12613 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:47.269634 12613 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:47.276333 12613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Generated new TSK 0
I20260929 00:49:47.277161 12613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:49:47.320570 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33345
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:47.455225 12630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:47.455510 12630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:47.455556 12630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:47.460171 12630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:47.460260 12630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:47.460304 12630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:47.460384 12630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:49:47.466153 12630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:47.467366 12630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:47.468442 12630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:47.478946 12635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.478965 12638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:47.479019 12630 server_base.cc:1034] running on GCE node
W20260929 00:49:47.478982 12636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:47.479616 12630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:47.480259 12630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:47.481599 12630 hybrid_clock.cc:648] HybridClock initialized: now 1790642987481559 us; error 53 us; skew 500 ppm
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:47.485297 12630 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:49:47.486631 12630 webserver.cc:460] Webserver started at http://127.11.40.193:38253/ using document root <none> and password file <none>
I20260929 00:49:47.487416 12630 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:47.487491 12630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:47.487741 12630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:47.490097 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "31cd02ada7d5b56357dace7f589e433e"
server_key_iv: "21b73576c41d81d620edbff563a9fbfb"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.490798 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "31cd02ada7d5b56357dace7f589e433e"
server_key_iv: "21b73576c41d81d620edbff563a9fbfb"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.495261 12630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:49:47.498076 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.499466 12630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:49:47.499598 12630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "31cd02ada7d5b56357dace7f589e433e"
server_key_iv: "21b73576c41d81d620edbff563a9fbfb"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.499720 12630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:47.526680 12630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:47.530309 12630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:47.530535 12630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:47.532068 12630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:47.532119 12630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.532158 12630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:47.532215 12630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:47.543761 12630 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:38193
I20260929 00:49:47.544708 12630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:49:47.546078 12760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:38193 every 8 connection(s)
I20260929 00:49:47.548905 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12630
I20260929 00:49:47.549026 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:49:47.549368 11427 external_mini_cluster.cc:1428] Setting key 1be728878dff9f497df0e45572b46914
I20260929 00:49:47.552469 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.533274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:50511 (local address 127.11.40.254:33345)
0929 00:49:47.533432 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:47.533438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:47.534321 (+ 883us) server_negotiation.cc:407] Connection header received
0929 00:49:47.535768 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:47.535773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:47.535839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:47.535939 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:47.537309 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.537889 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.539188 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.539437 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.539555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.543864 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:47.543869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:47.543873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:47.543915 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:47.549767 (+ 5852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.550506 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.550510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.550512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.550615 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.551074 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.551077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.551078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.551321 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:47.551450 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:47.552020 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 00:49:47.552180 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:49:47.553453 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.533670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.193:50511)
0929 00:49:47.534155 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:47.534214 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:47.535500 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:47.536126 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:47.536133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:47.536597 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:47.537150 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.537166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.538026 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.538031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.539039 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.539052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.539611 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.539614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.539790 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.541439 (+ 1649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:47.541460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:47.543605 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:47.549920 (+ 6315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.549928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.549948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.550354 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.550730 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.550734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.550735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.550957 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.551479 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:47.551485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:47.551756 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 00:49:47.551957 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 00:49:47.552252 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260929 00:49:47.554980 12761 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33345
I20260929 00:49:47.555258 12761 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:47.555791 12761 heartbeater.cc:511] Master 127.11.40.254:33345 requested a full tablet report, sending...
I20260929 00:49:47.557497 12552 ts_manager.cc:194] Registered new tserver with Master: 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193)
I20260929 00:49:47.559136 12552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:50511
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:49:47.567261 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.558010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.193:37563)
0929 00:49:47.558484 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:47.558501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:47.558654 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:47.559073 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:47.559077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:47.559297 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:47.559823 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.559837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.561059 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.561062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.561912 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.561921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.562433 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.562436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.562513 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.563039 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:47.563051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:47.563396 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:47.565595 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.565599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.565601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.565824 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.566177 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.566179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.566180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.566258 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.566776 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:47.566779 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:47.566875 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 00:49:47.566970 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 00:49:47.567097 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260929 00:49:47.567440 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.558074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37563 (local address 127.11.40.254:33345)
0929 00:49:47.558190 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:47.558195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:47.558575 (+ 380us) server_negotiation.cc:407] Connection header received
0929 00:49:47.558771 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:47.558776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:47.558834 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:47.558894 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:47.560006 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.560885 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.562025 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.562256 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.562342 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.563533 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:47.563536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:47.563538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:47.563561 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:47.565449 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.565955 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.565962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.565966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.566070 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.566351 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.566357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.566358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.566635 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:47.566746 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:47.567058 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 00:49:47.567205 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:49:47.584543 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33345
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:47.699790 12770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:47.700201 12770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:47.700251 12770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:47.704768 12770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:47.704866 12770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:47.704910 12770 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:47.704973 12770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:49:47.710680 12770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:47.712366 12770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:47.713814 12770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:47.724387 12776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.724512 12775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.724393 12778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:47.724740 12770 server_base.cc:1034] running on GCE node
I20260929 00:49:47.725286 12770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:47.725965 12770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:47.727188 12770 hybrid_clock.cc:648] HybridClock initialized: now 1790642987727129 us; error 72 us; skew 500 ppm
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:47.730767 12770 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:49:47.731827 12770 webserver.cc:460] Webserver started at http://127.11.40.194:36615/ using document root <none> and password file <none>
I20260929 00:49:47.732390 12770 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:47.732461 12770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:47.732632 12770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:47.734941 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "b666dd1ec19b40a4adeec2b64f60b912"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "19e7394f4acecaceba97f671f8ce7096"
server_key_iv: "042f523ab4a73aa162783edfc461a7ad"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.735474 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "b666dd1ec19b40a4adeec2b64f60b912"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "19e7394f4acecaceba97f671f8ce7096"
server_key_iv: "042f523ab4a73aa162783edfc461a7ad"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.739400 12770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:49:47.742221 12785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.743717 12770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:49:47.743837 12770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "b666dd1ec19b40a4adeec2b64f60b912"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "19e7394f4acecaceba97f671f8ce7096"
server_key_iv: "042f523ab4a73aa162783edfc461a7ad"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.743935 12770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:47.758723 12770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:47.762156 12770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:47.762315 12770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:47.763744 12770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:47.763832 12770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.763897 12770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:47.763947 12770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:47.774864 12770 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:43783
I20260929 00:49:47.774842 12900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:43783 every 8 connection(s)
I20260929 00:49:47.775990 12770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:49:47.779562 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.764989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56215 (local address 127.11.40.254:33345)
0929 00:49:47.765164 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:47.765170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:47.766246 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 00:49:47.767750 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:47.767754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:47.767810 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:47.767978 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:47.769904 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.770612 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.772061 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.772306 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.772438 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.775601 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:47.775608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:47.775614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:47.775644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:47.777398 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.777965 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.777968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.777970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.778074 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.778433 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.778436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.778438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.778639 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:47.778746 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:47.779258 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 00:49:47.779388 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:49:47.780465 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.765510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.194:56215)
0929 00:49:47.766036 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:47.766109 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:47.767455 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:47.768222 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:47.768235 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:47.768866 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:47.769698 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.769727 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.770763 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.770767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.771926 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.771936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.772483 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.772488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.772726 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.773452 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:47.773472 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:47.775383 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:47.777520 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.777526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.777549 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.777837 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.778171 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.778175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.778177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.778325 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.778756 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:47.778764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:47.779014 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 00:49:47.779206 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 00:49:47.779535 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260929 00:49:47.781482 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12770
I20260929 00:49:47.781606 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:49:47.781942 11427 external_mini_cluster.cc:1428] Setting key 33cd136560e4e0e490bddc5bd2e45abc
I20260929 00:49:47.782186 12901 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33345
I20260929 00:49:47.782356 12901 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:47.783643 12901 heartbeater.cc:511] Master 127.11.40.254:33345 requested a full tablet report, sending...
I20260929 00:49:47.785010 12558 ts_manager.cc:194] Registered new tserver with Master: b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783)
I20260929 00:49:47.786072 12558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:56215
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:49:47.798394 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.784394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.194:47383)
0929 00:49:47.784877 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:47.784895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:47.785035 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:47.785641 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:47.785644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:47.786174 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:47.786335 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.786342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.787768 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.787772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.789124 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:47.789293 (+ 169us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.790110 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.790118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:47.790554 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.791636 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:47.791661 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:47.792622 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:47.795737 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.795741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.795743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.796038 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.796599 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:47.796603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:47.796606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:47.796708 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:47.797592 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:47.797598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:47.797731 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 00:49:47.797859 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 00:49:47.798141 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260929 00:49:47.798568 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:47.784446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:47383 (local address 127.11.40.254:33345)
0929 00:49:47.784569 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:47.784573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:47.784986 (+ 413us) server_negotiation.cc:407] Connection header received
0929 00:49:47.785283 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:47.785296 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:47.785347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:47.785626 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:47.786495 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.787587 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.789444 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:47.789886 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:47.790026 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:47.792986 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:47.792992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:47.792996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:47.793034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:47.795575 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.796195 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.796200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.796202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.796447 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:47.796857 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:47.796864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:47.796868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:47.797468 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:47.797598 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:47.798025 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 00:49:47.798276 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:49:47.820708 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33345
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:47.936290 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:47.936615 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:47.936714 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:47.940908 12910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:47.940981 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:47.941021 12910 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:47.941056 12910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:49:47.946403 12910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:47.947640 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:47.948711 12910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:47.959965 12918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.959946 12915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:47.960003 12916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:47.960547 12910 server_base.cc:1034] running on GCE node
I20260929 00:49:47.961122 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:47.961812 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:47.963060 12910 hybrid_clock.cc:648] HybridClock initialized: now 1790642987963039 us; error 32 us; skew 500 ppm
Sep 29 00:49:47 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:47.966671 12910 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:49:47.967720 12910 webserver.cc:460] Webserver started at http://127.11.40.195:38977/ using document root <none> and password file <none>
I20260929 00:49:47.968323 12910 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:47.968428 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:47.968672 12910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:47.970922 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "a5f1209ba3964a42b622ecb5d8486d79"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "4bdac8450fb828545726439a16d752b6"
server_key_iv: "8c7e72442f63aa50292ca9b30bf2537f"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.971541 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "a5f1209ba3964a42b622ecb5d8486d79"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "4bdac8450fb828545726439a16d752b6"
server_key_iv: "8c7e72442f63aa50292ca9b30bf2537f"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.975878 12910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:47.978644 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:47.980022 12910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:49:47.980166 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "a5f1209ba3964a42b622ecb5d8486d79"
format_stamp: "Formatted at 2026-09-29 00:49:47 on dist-test-slave-5309"
server_key: "4bdac8450fb828545726439a16d752b6"
server_key_iv: "8c7e72442f63aa50292ca9b30bf2537f"
server_key_version: "encryptionkey@0"
I20260929 00:49:47.980294 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:47.999007 12910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:48.002717 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:48.002976 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:48.004691 12910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:48.004766 12910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:48.004896 12910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:48.004951 12910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:48.017699 12910 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:40927
I20260929 00:49:48.018774 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:49:48.019678 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:40927 every 8 connection(s)
I20260929 00:49:48.020720 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.005745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:42249 (local address 127.11.40.254:33345)
0929 00:49:48.005916 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.005921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.007078 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 00:49:48.008538 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.008543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.008616 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.008721 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.010166 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.010722 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.012029 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.012272 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.012390 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.016442 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.016447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.016450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.016480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.018327 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.018981 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.018985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.018987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.019090 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.019508 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.019511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.019513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.019758 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.019902 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.020473 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.020594 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:49:48.021615 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.006252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.195:42249)
0929 00:49:48.006797 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.006863 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.008172 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.008957 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.008967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.009460 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:48.009998 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.010013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.010859 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.010870 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.011899 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.011908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.012440 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.012445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.012645 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.013374 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:48.013396 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:48.016204 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:48.018465 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.018471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.018486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.018836 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.019202 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.019208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.019211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.019382 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.019950 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:48.019955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:48.020184 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.020346 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.020611 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260929 00:49:48.023201 13041 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33345
I20260929 00:49:48.023387 13041 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:48.024052 13041 heartbeater.cc:511] Master 127.11.40.254:33345 requested a full tablet report, sending...
I20260929 00:49:48.025207 12558 ts_manager.cc:194] Registered new tserver with Master: a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927)
I20260929 00:49:48.025897 12558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:42249
I20260929 00:49:48.028486 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 12910
I20260929 00:49:48.028666 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:49:48.029028 11427 external_mini_cluster.cc:1428] Setting key 61f0e26f2592027e7d0c69b03cfd789c
I20260929 00:49:48.031441 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:49:48.033857 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.025410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.11.40.195:52123)
0929 00:49:48.025766 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.025777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.025897 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.026169 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.026172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.026388 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:48.026487 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.026494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.027280 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.027283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.027945 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.027956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.028944 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.028947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.029040 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.029685 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:48.029702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:48.030199 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:48.032064 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.032070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.032074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.032358 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.032658 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.032663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.032668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.032775 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.033283 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:48.033289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:48.033419 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.033560 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.033707 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260929 00:49:48.034041 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.025484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52123 (local address 127.11.40.254:33345)
0929 00:49:48.025630 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.025636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.025851 (+ 215us) server_negotiation.cc:407] Connection header received
0929 00:49:48.026004 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.026008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.026062 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.026151 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.026608 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.027125 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.028085 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.028450 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.028589 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.030341 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.030345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.030348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.030384 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.031951 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.032470 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.032473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.032474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.032564 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.032955 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.032959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.032960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.033171 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.033274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.033651 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.033820 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[12513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642988, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642988, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:48.060767 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.051775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.11.40.254:33345)
0929 00:49:48.051957 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.051961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.051977 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:49:48.052024 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.052026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.052068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.052150 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.052997 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.053597 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.054485 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.054723 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.054814 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.057453 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.057456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.057458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.057486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.059072 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.059473 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.059476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.059477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.059554 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.059911 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.059914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.059915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.060126 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.060226 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.060543 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.060644 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:49:48.064689 12558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:49:48.067977 12558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:48.084705 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.077500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:40927 (local address 127.0.0.1:59604)
0929 00:49:48.078118 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.078174 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.078303 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.079780 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.079786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.079810 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.080009 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.080016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081227 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.081233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.082713 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.082723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.084002 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.084008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.084133 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084173 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.084315 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.084456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260929 00:49:48.085096 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.077861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.11.40.193:38193)
0929 00:49:48.078602 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.078614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.079011 (+ 397us) server_negotiation.cc:407] Connection header received
0929 00:49:48.079104 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.079110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.079362 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.079516 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.080163 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081496 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.083137 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.083952 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.084093 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084766 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.084819 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.084941 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":90,"threads_started":1}
I20260929 00:49:48.084905 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.077692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38193 (local address 127.0.0.1:45840)
0929 00:49:48.078895 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.078906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.078997 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.079800 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.079803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.079822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.080026 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.080036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081617 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.081620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.082990 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.083002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.084407 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.084410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.084479 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084501 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.084590 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.084665 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":52,"thread_start_us":132,"threads_started":1}
I20260929 00:49:48.084719 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.078314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43783 (local address 127.0.0.1:47142)
0929 00:49:48.079293 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.079306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.079393 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.080083 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.080089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.080112 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.080195 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.080200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081610 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.081613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.083101 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.083111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.084360 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.084363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.084460 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084480 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.084554 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.084615 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":183,"thread_start_us":87,"threads_started":1}
I20260929 00:49:48.085485 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.078500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.11.40.194:43783)
0929 00:49:48.078847 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.078865 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.079493 (+ 628us) server_negotiation.cc:407] Connection header received
0929 00:49:48.079680 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.079686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.079907 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.080090 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.080308 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081470 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.083225 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.084019 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.084127 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084286 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.084615 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.084724 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":121,"threads_started":1}
I20260929 00:49:48.085891 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.078609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.11.40.195:40927)
0929 00:49:48.079263 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.079276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.079302 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:49:48.079427 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.079442 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.079619 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.079784 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.080187 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.081109 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.082844 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.083815 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.083937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.084154 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.085610 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.085720 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":142,"threads_started":1}
I20260929 00:49:48.088258 12835 tablet_service.cc:1467] Processing CreateTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 (DEFAULT_TABLE table=test-table [id=07119d9bb2c14277bb0b06967d1a835b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.088265 12975 tablet_service.cc:1467] Processing CreateTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 (DEFAULT_TABLE table=test-table [id=07119d9bb2c14277bb0b06967d1a835b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.088254 12695 tablet_service.cc:1467] Processing CreateTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 (DEFAULT_TABLE table=test-table [id=07119d9bb2c14277bb0b06967d1a835b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.089421 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afe0fc64c98449f99ed8562fd70ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.089425 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afe0fc64c98449f99ed8562fd70ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.089426 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afe0fc64c98449f99ed8562fd70ff73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.096549 13064 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Bootstrap starting.
I20260929 00:49:48.097491 13066 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Bootstrap starting.
I20260929 00:49:48.098814 13064 tablet_bootstrap.cc:654] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.099213 13066 tablet_bootstrap.cc:654] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.099640 13064 log.cc:824] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:48.099995 13066 log.cc:824] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:48.100435 13065 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Bootstrap starting.
I20260929 00:49:48.101781 13064 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: No bootstrap required, opened a new log
I20260929 00:49:48.102006 13064 ts_tablet_manager.cc:1397] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:49:48.102093 13066 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: No bootstrap required, opened a new log
I20260929 00:49:48.102283 13066 ts_tablet_manager.cc:1397] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 00:49:48.102682 13065 tablet_bootstrap.cc:654] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.103538 13065 log.cc:824] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:48.105894 13065 tablet_bootstrap.cc:492] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: No bootstrap required, opened a new log
I20260929 00:49:48.105762 13064 raft_consensus.cc:348] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.106017 13064 raft_consensus.cc:374] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.106072 13064 raft_consensus.cc:729] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b666dd1ec19b40a4adeec2b64f60b912, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.106119 13065 ts_tablet_manager.cc:1397] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:49:48.106549 13064 consensus_queue.cc:260] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.106468 13066 raft_consensus.cc:348] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.107007 13066 raft_consensus.cc:374] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.107064 13066 raft_consensus.cc:729] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f1209ba3964a42b622ecb5d8486d79, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.107646 13064 ts_tablet_manager.cc:1428] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:49:48.107549 13066 consensus_queue.cc:260] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.108011 12901 heartbeater.cc:503] Master 127.11.40.254:33345 was elected leader, sending a full tablet report...
I20260929 00:49:48.108605 13066 ts_tablet_manager.cc:1428] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:49:48.108994 13041 heartbeater.cc:503] Master 127.11.40.254:33345 was elected leader, sending a full tablet report...
I20260929 00:49:48.110841 13065 raft_consensus.cc:348] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.111167 13065 raft_consensus.cc:374] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.111236 13065 raft_consensus.cc:729] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b87b1302c0340a3b2ef78a3f45bb5d8, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.111740 13065 consensus_queue.cc:260] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.112432 12761 heartbeater.cc:503] Master 127.11.40.254:33345 was elected leader, sending a full tablet report...
I20260929 00:49:48.112874 13065 ts_tablet_manager.cc:1428] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 00:49:48.207253 13071 raft_consensus.cc:482] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:48.207492 13071 raft_consensus.cc:504] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.208879 13071 leader_election.cc:288] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193), b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783)
I20260929 00:49:48.213374 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.209196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43783 (local address 127.11.40.195:44767)
0929 00:49:48.209410 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.209426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.209541 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.209882 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.209885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.209905 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.210048 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.210058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.211110 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.211116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.211925 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.211963 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.212875 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.212881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.213034 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.213055 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.213150 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.213231 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 00:49:48.213374 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.209284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:44767 (local address 127.11.40.194:43783)
0929 00:49:48.209412 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.209417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.209508 (+ 91us) server_negotiation.cc:407] Connection header received
0929 00:49:48.209662 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.209666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.209712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.209799 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.210168 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.211017 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.212093 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.212723 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.212870 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.212923 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.213171 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.213248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:49:48.214135 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afe0fc64c98449f99ed8562fd70ff73" candidate_uuid: "a5f1209ba3964a42b622ecb5d8486d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b666dd1ec19b40a4adeec2b64f60b912" is_pre_election: true
I20260929 00:49:48.214246 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.209196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38193 (local address 127.11.40.195:41775)
0929 00:49:48.209907 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.209920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.210017 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.210267 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.210270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.210289 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.210945 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.210956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.212009 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.212014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.212807 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.212821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.213839 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.213842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.213944 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.213965 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.214036 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.214101 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":311,"thread_start_us":157,"threads_started":1}
I20260929 00:49:48.214308 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.209373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:41775 (local address 127.11.40.193:38193)
0929 00:49:48.209495 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.209500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.210016 (+ 516us) server_negotiation.cc:407] Connection header received
0929 00:49:48.210112 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.210115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.210159 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.210231 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.211090 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.211869 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.212991 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.213599 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.213717 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.213762 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.214092 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.214189 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:49:48.214491 12855 raft_consensus.cc:2454] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f1209ba3964a42b622ecb5d8486d79 in term 0.
I20260929 00:49:48.215049 12928 leader_election.cc:302] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f1209ba3964a42b622ecb5d8486d79, b666dd1ec19b40a4adeec2b64f60b912; no voters:
I20260929 00:49:48.215363 13071 raft_consensus.cc:2781] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:48.215317 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afe0fc64c98449f99ed8562fd70ff73" candidate_uuid: "a5f1209ba3964a42b622ecb5d8486d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" is_pre_election: true
I20260929 00:49:48.215473 13071 raft_consensus.cc:482] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:48.215531 13071 raft_consensus.cc:3037] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.215595 12715 raft_consensus.cc:2454] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f1209ba3964a42b622ecb5d8486d79 in term 0.
I20260929 00:49:48.216725 13071 raft_consensus.cc:504] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.217164 13071 leader_election.cc:288] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [CANDIDATE]: Term 1 election: Requested vote from peers 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193), b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783)
I20260929 00:49:48.217518 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afe0fc64c98449f99ed8562fd70ff73" candidate_uuid: "a5f1209ba3964a42b622ecb5d8486d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8"
I20260929 00:49:48.217557 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afe0fc64c98449f99ed8562fd70ff73" candidate_uuid: "a5f1209ba3964a42b622ecb5d8486d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b666dd1ec19b40a4adeec2b64f60b912"
I20260929 00:49:48.217675 12855 raft_consensus.cc:3037] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.217681 12715 raft_consensus.cc:3037] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.218773 12715 raft_consensus.cc:2454] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f1209ba3964a42b622ecb5d8486d79 in term 1.
I20260929 00:49:48.218964 12855 raft_consensus.cc:2454] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f1209ba3964a42b622ecb5d8486d79 in term 1.
I20260929 00:49:48.219189 12929 leader_election.cc:302] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b87b1302c0340a3b2ef78a3f45bb5d8, a5f1209ba3964a42b622ecb5d8486d79; no voters:
I20260929 00:49:48.219426 13071 raft_consensus.cc:2781] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:48.219692 13071 raft_consensus.cc:686] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 LEADER]: Becoming Leader. State: Replica: a5f1209ba3964a42b622ecb5d8486d79, State: Running, Role: LEADER
I20260929 00:49:48.220072 13071 consensus_queue.cc:237] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } }
I20260929 00:49:48.223879 12552 catalog_manager.cc:5495] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "a5f1209ba3964a42b622ecb5d8486d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260929 00:49:48.245569 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.241410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.11.40.195:40927)
0929 00:49:48.241612 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.241616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.241632 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:49:48.241892 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.241894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.241951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.242049 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:48.242309 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.242881 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.243886 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.244132 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.244227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.244580 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.245135 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.245229 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.245379 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.245439 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:49:48.252060 12715 raft_consensus.cc:1261] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Refusing update from remote peer a5f1209ba3964a42b622ecb5d8486d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.252056 12855 raft_consensus.cc:1261] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Refusing update from remote peer a5f1209ba3964a42b622ecb5d8486d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.252868 13074 consensus_queue.cc:1035] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:48.253085 13071 consensus_queue.cc:1035] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:48.262084 13077 mvcc.cc:205] Tried to move back new op lower bound from 7334473679872987136 to 7334473679752740864. Current Snapshot: MvccSnapshot[applied={T|T < 7334473679872987136}]
I20260929 00:49:48.265563 13078 mvcc.cc:205] Tried to move back new op lower bound from 7334473679872987136 to 7334473679752740864. Current Snapshot: MvccSnapshot[applied={T|T < 7334473679872987136}]
I20260929 00:49:48.276345 12552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:48.276563 12552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:48.279721 12552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Sending DeleteTablet for 3 replicas of tablet 3afe0fc64c98449f99ed8562fd70ff73
I20260929 00:49:48.280683 12695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:45840
I20260929 00:49:48.280995 12975 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59604
I20260929 00:49:48.281127 12835 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afe0fc64c98449f99ed8562fd70ff73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:47142
I20260929 00:49:48.281663 13087 tablet_replica.cc:330] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: stopping tablet replica
I20260929 00:49:48.281682 12552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:49:48.281881 13088 tablet_replica.cc:330] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: stopping tablet replica
I20260929 00:49:48.282096 13087 raft_consensus.cc:2229] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 00:49:48.282109 12552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:48.282159 13088 raft_consensus.cc:2229] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:48.282419 13087 raft_consensus.cc:2258] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.282465 13088 raft_consensus.cc:2258] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.283917 13088 ts_tablet_manager.cc:1905] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.283917 13087 ts_tablet_manager.cc:1905] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.286725 13088 ts_tablet_manager.cc:1918] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:49:48.286846 13088 log.cc:1196] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/3afe0fc64c98449f99ed8562fd70ff73
I20260929 00:49:48.287158 13088 ts_tablet_manager.cc:1939] T 3afe0fc64c98449f99ed8562fd70ff73 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting consensus metadata
I20260929 00:49:48.288595 12545 catalog_manager.cc:4841] TS 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193): tablet 3afe0fc64c98449f99ed8562fd70ff73 (table test-table [id=07119d9bb2c14277bb0b06967d1a835b]) successfully deleted
I20260929 00:49:48.289023 13087 ts_tablet_manager.cc:1918] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:49:48.289142 13087 log.cc:1196] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/3afe0fc64c98449f99ed8562fd70ff73
I20260929 00:49:48.289289 13089 tablet_replica.cc:330] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: stopping tablet replica
I20260929 00:49:48.289462 13087 ts_tablet_manager.cc:1939] T 3afe0fc64c98449f99ed8562fd70ff73 P b666dd1ec19b40a4adeec2b64f60b912: Deleting consensus metadata
I20260929 00:49:48.289719 13089 raft_consensus.cc:2229] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:48.290644 12544 catalog_manager.cc:4841] TS b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783): tablet 3afe0fc64c98449f99ed8562fd70ff73 (table test-table [id=07119d9bb2c14277bb0b06967d1a835b]) successfully deleted
I20260929 00:49:48.291224 13089 raft_consensus.cc:2258] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.292629 13089 ts_tablet_manager.cc:1905] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.293188 12835 tablet_service.cc:1467] Processing CreateTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 (DEFAULT_TABLE table=test-table [id=e0f46bfe0d07471c9fe9fc194500b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.293254 12975 tablet_service.cc:1467] Processing CreateTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 (DEFAULT_TABLE table=test-table [id=e0f46bfe0d07471c9fe9fc194500b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.293619 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf4aedc4dd54bc3963b06fee77a3252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.294016 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf4aedc4dd54bc3963b06fee77a3252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.294230 12695 tablet_service.cc:1467] Processing CreateTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 (DEFAULT_TABLE table=test-table [id=e0f46bfe0d07471c9fe9fc194500b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:48.294461 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf4aedc4dd54bc3963b06fee77a3252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.296787 13066 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Bootstrap starting.
I20260929 00:49:48.297010 13064 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Bootstrap starting.
I20260929 00:49:48.298192 13066 tablet_bootstrap.cc:654] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.298688 13064 tablet_bootstrap.cc:654] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.300087 13066 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: No bootstrap required, opened a new log
I20260929 00:49:48.300206 13066 ts_tablet_manager.cc:1397] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260929 00:49:48.300558 13064 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: No bootstrap required, opened a new log
I20260929 00:49:48.300639 13064 ts_tablet_manager.cc:1397] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 00:49:48.300976 13066 raft_consensus.cc:348] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.301100 13066 raft_consensus.cc:374] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.301141 13066 raft_consensus.cc:729] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f1209ba3964a42b622ecb5d8486d79, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.301285 13064 raft_consensus.cc:348] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.301311 13066 consensus_queue.cc:260] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.301409 13064 raft_consensus.cc:374] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.301442 13064 raft_consensus.cc:729] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b666dd1ec19b40a4adeec2b64f60b912, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.301695 13066 ts_tablet_manager.cc:1428] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 00:49:48.302045 13065 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Bootstrap starting.
I20260929 00:49:48.303686 13065 tablet_bootstrap.cc:654] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.303957 13064 consensus_queue.cc:260] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.305028 13065 tablet_bootstrap.cc:492] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: No bootstrap required, opened a new log
I20260929 00:49:48.305120 13065 ts_tablet_manager.cc:1397] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:48.305635 13065 raft_consensus.cc:348] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.305753 13065 raft_consensus.cc:374] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.305756 13064 ts_tablet_manager.cc:1428] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 00:49:48.305794 13065 raft_consensus.cc:729] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b87b1302c0340a3b2ef78a3f45bb5d8, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.305938 13065 consensus_queue.cc:260] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.306205 13065 ts_tablet_manager.cc:1428] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:49:48.310811 13089 ts_tablet_manager.cc:1918] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:49:48.310954 13089 log.cc:1196] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/3afe0fc64c98449f99ed8562fd70ff73
I20260929 00:49:48.311271 13089 ts_tablet_manager.cc:1939] T 3afe0fc64c98449f99ed8562fd70ff73 P a5f1209ba3964a42b622ecb5d8486d79: Deleting consensus metadata
I20260929 00:49:48.312556 12544 catalog_manager.cc:4841] TS a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927): tablet 3afe0fc64c98449f99ed8562fd70ff73 (table test-table [id=07119d9bb2c14277bb0b06967d1a835b]) successfully deleted
I20260929 00:49:48.361790 13070 raft_consensus.cc:482] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:48.361955 13070 raft_consensus.cc:504] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.363104 13070 leader_election.cc:288] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193), a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927)
I20260929 00:49:48.368265 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.364201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56625 (local address 127.11.40.193:38193)
0929 00:49:48.364338 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.364341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.364352 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:49:48.364396 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.364397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.364441 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.364536 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.364827 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.365524 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.366907 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.367558 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.367705 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.367761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.368052 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.368139 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:49:48.368294 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.363477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:58433 (local address 127.11.40.195:40927)
0929 00:49:48.363609 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.363614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.363755 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:49:48.364017 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.364019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.364064 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.364163 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.364528 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.365335 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.367049 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.367641 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.367741 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.367793 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.368102 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.368168 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:49:48.368366 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.363343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38193 (local address 127.11.40.194:56625)
0929 00:49:48.363782 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.363793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.363856 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.364553 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.364556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.364574 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.364683 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.364689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.365632 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.365638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.366771 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.366782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.367697 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.367701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.367898 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.367925 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.368013 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.368096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260929 00:49:48.368943 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.363333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:40927 (local address 127.11.40.194:58433)
0929 00:49:48.363671 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.363687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.363843 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.364214 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.364218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.364245 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.364386 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.364394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.365460 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.365464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.366902 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.366914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.367864 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.367867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.367940 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.367957 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.368713 (+ 756us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.368794 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":30,"thread_start_us":75,"threads_started":1}
I20260929 00:49:48.368875 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf4aedc4dd54bc3963b06fee77a3252" candidate_uuid: "b666dd1ec19b40a4adeec2b64f60b912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" is_pre_election: true
I20260929 00:49:48.369048 12715 raft_consensus.cc:2454] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b666dd1ec19b40a4adeec2b64f60b912 in term 0.
I20260929 00:49:48.369536 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf4aedc4dd54bc3963b06fee77a3252" candidate_uuid: "b666dd1ec19b40a4adeec2b64f60b912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f1209ba3964a42b622ecb5d8486d79" is_pre_election: true
I20260929 00:49:48.369530 12789 leader_election.cc:302] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b87b1302c0340a3b2ef78a3f45bb5d8, b666dd1ec19b40a4adeec2b64f60b912; no voters:
I20260929 00:49:48.369841 13070 raft_consensus.cc:2781] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:48.369782 12995 raft_consensus.cc:2454] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b666dd1ec19b40a4adeec2b64f60b912 in term 0.
I20260929 00:49:48.369907 13070 raft_consensus.cc:482] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:48.369946 13070 raft_consensus.cc:3037] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.371284 13070 raft_consensus.cc:504] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.371680 13070 leader_election.cc:288] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [CANDIDATE]: Term 1 election: Requested vote from peers 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193), a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927)
I20260929 00:49:48.372035 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf4aedc4dd54bc3963b06fee77a3252" candidate_uuid: "b666dd1ec19b40a4adeec2b64f60b912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f1209ba3964a42b622ecb5d8486d79"
I20260929 00:49:48.372037 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf4aedc4dd54bc3963b06fee77a3252" candidate_uuid: "b666dd1ec19b40a4adeec2b64f60b912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8"
I20260929 00:49:48.372164 12715 raft_consensus.cc:3037] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.372169 12995 raft_consensus.cc:3037] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.373369 12995 raft_consensus.cc:2454] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b666dd1ec19b40a4adeec2b64f60b912 in term 1.
I20260929 00:49:48.373656 12715 raft_consensus.cc:2454] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b666dd1ec19b40a4adeec2b64f60b912 in term 1.
I20260929 00:49:48.373699 12788 leader_election.cc:302] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f1209ba3964a42b622ecb5d8486d79, b666dd1ec19b40a4adeec2b64f60b912; no voters:
I20260929 00:49:48.373983 13070 raft_consensus.cc:2781] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:48.374274 13070 raft_consensus.cc:686] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 LEADER]: Becoming Leader. State: Replica: b666dd1ec19b40a4adeec2b64f60b912, State: Running, Role: LEADER
I20260929 00:49:48.374539 13070 consensus_queue.cc:237] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.377252 12558 catalog_manager.cc:5495] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 reported cstate change: term changed from 0 to 1, leader changed from <none> to b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194). New cstate: current_term: 1 leader_uuid: "b666dd1ec19b40a4adeec2b64f60b912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642987, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:48.429333 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.418972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33345 (local address 127.0.0.1:51230)
0929 00:49:48.419381 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.419401 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.419529 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.419837 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.419839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.420248 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:48.420402 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.420410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.421537 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.421540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.422119 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.422128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.422765 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.422770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.422843 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.423633 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:48.423644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:48.425309 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:48.427520 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.427533 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.427541 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.427828 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.428145 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:48.428148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:48.428149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:48.428257 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:48.428705 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:48.428711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:48.428884 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.429013 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.429177 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":176,"threads_started":1}
I20260929 00:49:48.429479 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.419122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.11.40.254:33345)
0929 00:49:48.419311 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.419316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.419503 (+ 187us) server_negotiation.cc:407] Connection header received
0929 00:49:48.419628 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.419632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.419685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.419799 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.420544 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.421403 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.422253 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.422585 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.422701 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.425478 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.425482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.425485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.425512 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.427250 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.427944 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.427948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.427950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.428044 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.428332 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.428335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.428337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.428602 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.428690 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.429089 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.429255 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":74}
I20260929 00:49:48.434007 12558 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:51230:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:49:48.434743 12558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:48.440227 12695 tablet_service.cc:1467] Processing CreateTablet for tablet f2b998fd60814549aa3a2b9e67b81b1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd884a34d5204480a85ca8b0b496d7c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:48.440400 12835 tablet_service.cc:1467] Processing CreateTablet for tablet f2b998fd60814549aa3a2b9e67b81b1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd884a34d5204480a85ca8b0b496d7c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:48.440474 12975 tablet_service.cc:1467] Processing CreateTablet for tablet f2b998fd60814549aa3a2b9e67b81b1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd884a34d5204480a85ca8b0b496d7c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:48.440680 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b998fd60814549aa3a2b9e67b81b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.440840 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b998fd60814549aa3a2b9e67b81b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.440840 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b998fd60814549aa3a2b9e67b81b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:48.444150 13066 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: Bootstrap starting.
I20260929 00:49:48.444205 13064 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: Bootstrap starting.
I20260929 00:49:48.445278 13066 tablet_bootstrap.cc:654] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.445394 13064 tablet_bootstrap.cc:654] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.446712 13065 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: Bootstrap starting.
I20260929 00:49:48.446817 13066 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: No bootstrap required, opened a new log
I20260929 00:49:48.446920 13066 ts_tablet_manager.cc:1397] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:48.447439 13066 raft_consensus.cc:348] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.447649 13066 raft_consensus.cc:374] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.447696 13066 raft_consensus.cc:729] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f1209ba3964a42b622ecb5d8486d79, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.447892 13065 tablet_bootstrap.cc:654] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:48.447868 13066 consensus_queue.cc:260] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.448230 13066 ts_tablet_manager.cc:1428] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:49:48.448226 13083 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.448376 13083 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.449358 13064 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: No bootstrap required, opened a new log
I20260929 00:49:48.449448 13064 ts_tablet_manager.cc:1397] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 00:49:48.449843 13064 raft_consensus.cc:348] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.449970 13064 raft_consensus.cc:374] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.449999 13064 raft_consensus.cc:729] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b666dd1ec19b40a4adeec2b64f60b912, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.450114 13064 consensus_queue.cc:260] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.450157 13065 tablet_bootstrap.cc:492] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: No bootstrap required, opened a new log
I20260929 00:49:48.450265 13065 ts_tablet_manager.cc:1397] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:49:48.450398 13070 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.450618 13070 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.451027 13065 raft_consensus.cc:348] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.451150 13065 raft_consensus.cc:374] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:48.451187 13065 raft_consensus.cc:729] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b87b1302c0340a3b2ef78a3f45bb5d8, State: Initialized, Role: FOLLOWER
I20260929 00:49:48.451319 13065 consensus_queue.cc:260] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.451610 13072 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.451761 13065 ts_tablet_manager.cc:1428] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:49:48.451771 13072 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.452791 13064 ts_tablet_manager.cc:1428] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 00:49:48.455591 13072 raft_consensus.cc:482] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:48.455716 13072 raft_consensus.cc:504] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.457002 13072 leader_election.cc:288] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783), a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927)
I20260929 00:49:48.461834 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.457221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43783 (local address 127.11.40.193:59711)
0929 00:49:48.457523 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.457537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.457665 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.458715 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.458718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.458738 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.458880 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.458887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.459693 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.459695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.460558 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.460566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.461349 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.461356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.461465 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.461490 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.461572 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.461672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":39,"thread_start_us":122,"threads_started":1}
I20260929 00:49:48.461827 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.457968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59711 (local address 127.11.40.194:43783)
0929 00:49:48.458171 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.458175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.458188 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:49:48.458234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.458237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.458279 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.458354 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.459020 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.459577 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.460691 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.461221 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.461324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.461410 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.461603 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.461700 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 00:49:48.461890 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.457962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:50863 (local address 127.11.40.195:40927)
0929 00:49:48.458081 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.458085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.458149 (+ 64us) server_negotiation.cc:407] Connection header received
0929 00:49:48.458297 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.458300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.458362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.458506 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:48.458953 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.459611 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.460895 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.461299 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.461418 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.461545 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.461702 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.461774 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:49:48.461921 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.457623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:40927 (local address 127.11.40.193:50863)
0929 00:49:48.457990 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.458005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.458189 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.458516 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.458519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.458538 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:48.458789 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.458799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.459738 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.459740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.460712 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.460722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.461431 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.461435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.461569 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.461594 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.461678 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.461743 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260929 00:49:48.462468 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b998fd60814549aa3a2b9e67b81b1a" candidate_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f1209ba3964a42b622ecb5d8486d79" is_pre_election: true
I20260929 00:49:48.462512 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b998fd60814549aa3a2b9e67b81b1a" candidate_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b666dd1ec19b40a4adeec2b64f60b912" is_pre_election: true
I20260929 00:49:48.462659 12995 raft_consensus.cc:2454] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b87b1302c0340a3b2ef78a3f45bb5d8 in term 0.
I20260929 00:49:48.462671 12855 raft_consensus.cc:2454] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b87b1302c0340a3b2ef78a3f45bb5d8 in term 0.
I20260929 00:49:48.463042 12648 leader_election.cc:302] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b87b1302c0340a3b2ef78a3f45bb5d8, a5f1209ba3964a42b622ecb5d8486d79; no voters:
I20260929 00:49:48.463325 13072 raft_consensus.cc:2781] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:48.463393 13072 raft_consensus.cc:482] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:48.463438 13072 raft_consensus.cc:3037] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.464497 13072 raft_consensus.cc:504] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.464885 13072 leader_election.cc:288] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [CANDIDATE]: Term 1 election: Requested vote from peers b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783), a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927)
I20260929 00:49:48.465166 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b998fd60814549aa3a2b9e67b81b1a" candidate_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b666dd1ec19b40a4adeec2b64f60b912"
I20260929 00:49:48.465233 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b998fd60814549aa3a2b9e67b81b1a" candidate_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f1209ba3964a42b622ecb5d8486d79"
I20260929 00:49:48.465293 12855 raft_consensus.cc:3037] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.465370 12995 raft_consensus.cc:3037] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:48.466253 12855 raft_consensus.cc:2454] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b87b1302c0340a3b2ef78a3f45bb5d8 in term 1.
I20260929 00:49:48.466293 12995 raft_consensus.cc:2454] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b87b1302c0340a3b2ef78a3f45bb5d8 in term 1.
I20260929 00:49:48.466652 12648 leader_election.cc:302] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b87b1302c0340a3b2ef78a3f45bb5d8, b666dd1ec19b40a4adeec2b64f60b912; no voters:
I20260929 00:49:48.466854 13072 raft_consensus.cc:2781] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:48.467095 13072 raft_consensus.cc:686] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 1b87b1302c0340a3b2ef78a3f45bb5d8, State: Running, Role: LEADER
I20260929 00:49:48.467357 13072 consensus_queue.cc:237] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } }
I20260929 00:49:48.468860 13108 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: TxnStatusTablet state changed. Reason: New leader 1b87b1302c0340a3b2ef78a3f45bb5d8. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.469003 13108 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:48.469262 13110 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:49:48.469416 13110 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:49:48.469861 12552 catalog_manager.cc:5495] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } health_report { overall_health: UNKNOWN } } }
I20260929 00:49:48.482940 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.478951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38193 (local address 127.0.0.1:45846)
0929 00:49:48.479132 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.479148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.479260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.479635 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.479638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.479646 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:48.479755 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.479761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.480499 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.480504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.481225 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.481234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.481727 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.481731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.481830 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.481888 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.482623 (+ 735us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.482642 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.482713 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.482808 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 00:49:48.482933 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.479134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.11.40.193:38193)
0929 00:49:48.479252 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.479256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.479273 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:49:48.479392 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.479395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.479510 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.479624 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:48.479872 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.480383 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.481356 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.481610 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.481741 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.482051 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.482523 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.482651 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.482722 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.482784 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:49:48.486570 12855 raft_consensus.cc:1261] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Refusing update from remote peer 1b87b1302c0340a3b2ef78a3f45bb5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.486796 12995 raft_consensus.cc:1261] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Refusing update from remote peer 1b87b1302c0340a3b2ef78a3f45bb5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.487318 13072 consensus_queue.cc:1035] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:48.487509 13108 consensus_queue.cc:1035] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:48.489133 13070 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: TxnStatusTablet state changed. Reason: New leader 1b87b1302c0340a3b2ef78a3f45bb5d8. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.489274 13070 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.491199 13083 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: TxnStatusTablet state changed. Reason: New leader 1b87b1302c0340a3b2ef78a3f45bb5d8. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.491341 13083 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.493295 13108 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.493440 13108 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:48.493547 13108 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.493611 13108 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P 1b87b1302c0340a3b2ef78a3f45bb5d8: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:48.493688 13079 mvcc.cc:205] Tried to move back new op lower bound from 7334473680834875392 to 7334473680765030400. Current Snapshot: MvccSnapshot[applied={T|T < 7334473680834875392 or (T in {7334473680834875392})}]
I20260929 00:49:48.495195 13083 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.495323 13083 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P a5f1209ba3964a42b622ecb5d8486d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.496598 13070 tablet_replica.cc:439] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 } } peers { permanent_uuid: "b666dd1ec19b40a4adeec2b64f60b912" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43783 } } peers { permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 } } }
I20260929 00:49:48.496740 13070 tablet_replica.cc:442] T f2b998fd60814549aa3a2b9e67b81b1a P b666dd1ec19b40a4adeec2b64f60b912: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:48.502986 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.498985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.11.40.194:43783)
0929 00:49:48.499095 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.499099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.499110 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:49:48.499171 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.499173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.499222 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.499309 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:48.499635 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.500133 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.501319 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.501615 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.501708 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.502024 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.502524 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.502644 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.502743 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.502796 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 00:49:48.515537 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.511630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:55449 (local address 127.11.40.193:38193)
0929 00:49:48.511767 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.511771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.512027 (+ 256us) server_negotiation.cc:407] Connection header received
0929 00:49:48.512200 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.512203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.512256 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.512341 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:48.512747 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.513306 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.514329 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.514556 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.514685 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.515017 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.515095 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.515176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.515296 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.515347 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:49:48.515841 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.511504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38193 (local address 127.11.40.194:55449)
0929 00:49:48.511936 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.511949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.512093 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.512352 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.512355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.512362 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:48.512593 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.512599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.513418 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.513423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.514201 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.514210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.514698 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.514702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.514830 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.514872 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.515181 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.515190 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.515273 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.515352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":158,"threads_started":1}
W20260929 00:49:48.520737 13042 tablet.cc:2369] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:49:48.523590 12995 raft_consensus.cc:1261] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Refusing update from remote peer b666dd1ec19b40a4adeec2b64f60b912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.523809 12715 raft_consensus.cc:1261] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Refusing update from remote peer b666dd1ec19b40a4adeec2b64f60b912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:48.524145 13095 consensus_queue.cc:1035] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f1209ba3964a42b622ecb5d8486d79" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 40927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:48.524331 13070 consensus_queue.cc:1035] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b87b1302c0340a3b2ef78a3f45bb5d8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:49:48.528649 12902 tablet.cc:2369] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:49:48.563194 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.558306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:50917 (local address 127.11.40.194:43783)
0929 00:49:48.558471 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.558475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.558872 (+ 397us) server_negotiation.cc:407] Connection header received
0929 00:49:48.559039 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.559043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.559095 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.559205 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:48.559820 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.560496 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.561871 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.562189 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.562273 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.562719 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.562802 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.562883 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.562976 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.563037 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:49:48.563230 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.558266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43783 (local address 127.11.40.193:50917)
0929 00:49:48.558777 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:48.558790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:48.558921 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:48.559319 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:48.559322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:48.559399 (+ 77us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:48.559689 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.559697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.560777 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.560784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.561736 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:48.561747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.562399 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.562402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:48.562498 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.562571 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:48.562899 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:48.562906 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:49:48.562967 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 00:49:48.563049 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":164,"threads_started":1}
I20260929 00:49:48.593353 12557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:48.593572 12557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:48.596251 12557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54865ce54f904f5fae86bead2f84d3db: Sending DeleteTablet for 3 replicas of tablet fcf4aedc4dd54bc3963b06fee77a3252
I20260929 00:49:48.597216 12695 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:45840
I20260929 00:49:48.597247 12975 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59604
I20260929 00:49:48.597473 13088 tablet_replica.cc:330] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: stopping tablet replica
I20260929 00:49:48.597470 13089 tablet_replica.cc:330] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: stopping tablet replica
I20260929 00:49:48.597455 12835 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf4aedc4dd54bc3963b06fee77a3252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:48 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:47142
I20260929 00:49:48.597609 13088 raft_consensus.cc:2229] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:48.597620 13089 raft_consensus.cc:2229] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:48.597663 13087 tablet_replica.cc:330] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: stopping tablet replica
I20260929 00:49:48.597738 13088 raft_consensus.cc:2258] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.597761 13089 raft_consensus.cc:2258] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.597770 13087 raft_consensus.cc:2229] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:48.598114 13087 raft_consensus.cc:2258] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:48.598814 13088 ts_tablet_manager.cc:1905] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.598973 13087 ts_tablet_manager.cc:1905] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.602056 13088 ts_tablet_manager.cc:1918] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:48.602056 13087 ts_tablet_manager.cc:1918] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:48.602145 13088 log.cc:1196] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/fcf4aedc4dd54bc3963b06fee77a3252
I20260929 00:49:48.602145 13087 log.cc:1196] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/fcf4aedc4dd54bc3963b06fee77a3252
I20260929 00:49:48.602417 13087 ts_tablet_manager.cc:1939] T fcf4aedc4dd54bc3963b06fee77a3252 P b666dd1ec19b40a4adeec2b64f60b912: Deleting consensus metadata
I20260929 00:49:48.602432 13088 ts_tablet_manager.cc:1939] T fcf4aedc4dd54bc3963b06fee77a3252 P 1b87b1302c0340a3b2ef78a3f45bb5d8: Deleting consensus metadata
I20260929 00:49:48.603139 13089 ts_tablet_manager.cc:1905] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:48.603717 12544 catalog_manager.cc:4841] TS b666dd1ec19b40a4adeec2b64f60b912 (127.11.40.194:43783): tablet fcf4aedc4dd54bc3963b06fee77a3252 (table test-table [id=e0f46bfe0d07471c9fe9fc194500b944]) successfully deleted
I20260929 00:49:48.606582 13089 ts_tablet_manager.cc:1918] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:48.606699 13089 log.cc:1196] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/fcf4aedc4dd54bc3963b06fee77a3252
I20260929 00:49:48.606952 13089 ts_tablet_manager.cc:1939] T fcf4aedc4dd54bc3963b06fee77a3252 P a5f1209ba3964a42b622ecb5d8486d79: Deleting consensus metadata
I20260929 00:49:48.607836 12545 catalog_manager.cc:4841] TS 1b87b1302c0340a3b2ef78a3f45bb5d8 (127.11.40.193:38193): tablet fcf4aedc4dd54bc3963b06fee77a3252 (table test-table [id=e0f46bfe0d07471c9fe9fc194500b944]) successfully deleted
I20260929 00:49:48.609104 12544 catalog_manager.cc:4841] TS a5f1209ba3964a42b622ecb5d8486d79 (127.11.40.195:40927): tablet fcf4aedc4dd54bc3963b06fee77a3252 (table test-table [id=e0f46bfe0d07471c9fe9fc194500b944]) successfully deleted
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[12513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642988, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:49:48.617807 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.601600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.11.40.193:38193)
0929 00:49:48.601743 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.601748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.602858 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 00:49:48.603049 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.603052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.603143 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.603276 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.604924 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.605921 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.607722 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.608066 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.608148 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.612866 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.612878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.613059 (+ 181us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.613110 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.615553 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.616197 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.616204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.616206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.616314 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.616657 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.616660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.616662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.617022 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.617223 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.617492 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.617630 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:49:48.618341 12740 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862
I20260929 00:49:48.630079 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:48.620199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.11.40.254:33345)
0929 00:49:48.620319 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:48.620323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:48.620744 (+ 421us) server_negotiation.cc:407] Connection header received
0929 00:49:48.620881 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:48.620884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:48.620979 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:48.621079 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:48.621887 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.622665 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.623820 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:48.624307 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:48.624418 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:48.626127 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:48.626131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:48.626134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:48.626166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:48.628013 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.628534 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.628538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.628540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.628635 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:48.629026 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:48.629030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:48.629032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:48.629331 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:48.629465 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:48.629736 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 00:49:48.629867 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 00:49:48.630473 12557 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51240
I20260929 00:49:48.632086 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12630
I20260929 00:49:48.639978 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12770
I20260929 00:49:48.646562 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12910
I20260929 00:49:48.653714 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 12526
2026-09-29T00:49:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2855 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[13132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[13132](info): set up 2 sockets
Sep 29 00:49:48 dist-test-slave-5309 krb5kdc[13132](info): commencing operation
krb5kdc: starting...
W20260929 00:49:49.706972 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:49 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:49.866734 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37681
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:49.990716 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:49.991051 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:49.991137 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:49.995327 13145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:49.995469 13145 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:49:49.995499 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:49.995682 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:49.995776 13145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:49.995822 13145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:50.001880 13145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:37681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:50.003177 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:50.004347 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:50.014472 13150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.014648 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.014824 13151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:50.014780 13145 server_base.cc:1034] running on GCE node
I20260929 00:49:50.015606 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:50.016889 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:50.018093 13145 hybrid_clock.cc:648] HybridClock initialized: now 1790642990018080 us; error 27 us; skew 500 ppm
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:50.021234 13145 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:50.022197 13145 webserver.cc:460] Webserver started at http://127.11.40.254:43869/ using document root <none> and password file <none>
I20260929 00:49:50.022785 13145 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:50.022879 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:50.023120 13145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:50.025307 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "4e664ad47ddb4ce5a22079f3b3669080"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "db10612cefab32d72e423e098296aefb"
server_key_iv: "756c666ef9c73dac03c5e370819ba6ea"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.025893 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "4e664ad47ddb4ce5a22079f3b3669080"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "db10612cefab32d72e423e098296aefb"
server_key_iv: "756c666ef9c73dac03c5e370819ba6ea"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.029809 13145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:49:50.032439 13160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.033914 13145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:49:50.034101 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "4e664ad47ddb4ce5a22079f3b3669080"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "db10612cefab32d72e423e098296aefb"
server_key_iv: "756c666ef9c73dac03c5e370819ba6ea"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.034235 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:50.055804 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:50.059584 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:50.059819 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:50.068876 13145 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:37681
I20260929 00:49:50.068928 13222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:37681 every 8 connection(s)
I20260929 00:49:50.069962 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:50.072891 13223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:50.076045 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13145
I20260929 00:49:50.076164 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:50.076442 11427 external_mini_cluster.cc:1428] Setting key f13a4b06c58118fd04681423a8bc84d1
I20260929 00:49:50.079092 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Bootstrap starting.
I20260929 00:49:50.081832 13223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:50.082626 13223 log.cc:824] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:50.084755 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: No bootstrap required, opened a new log
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:50.088627 13223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } }
I20260929 00:49:50.088944 13223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:50.089036 13223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e664ad47ddb4ce5a22079f3b3669080, State: Initialized, Role: FOLLOWER
I20260929 00:49:50.089573 13223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } }
I20260929 00:49:50.089767 13223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:50.089838 13223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:50.089952 13223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:50.091240 13223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } }
I20260929 00:49:50.091624 13223 leader_election.cc:302] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e664ad47ddb4ce5a22079f3b3669080; no voters:
I20260929 00:49:50.091966 13223 leader_election.cc:288] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:50.092101 13228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:50.092373 13228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [term 1 LEADER]: Becoming Leader. State: Replica: 4e664ad47ddb4ce5a22079f3b3669080, State: Running, Role: LEADER
I20260929 00:49:50.092414 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.077876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.11.40.254:37681)
0929 00:49:50.078331 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.078356 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.078393 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:49:50.079066 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.079084 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.079482 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.079771 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.080697 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.081941 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.082939 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.083667 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.083846 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.086777 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.086782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.086796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.086824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.088759 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.089466 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.089475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.089483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.089621 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.089969 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.089975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.089977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.090507 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.090727 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.091023 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.091353 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
I20260929 00:49:50.092712 13228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } }
I20260929 00:49:50.093358 13223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:50.094376 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e664ad47ddb4ce5a22079f3b3669080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } } }
I20260929 00:49:50.094513 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:50.094743 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e664ad47ddb4ce5a22079f3b3669080. Latest consensus state: current_term: 1 leader_uuid: "4e664ad47ddb4ce5a22079f3b3669080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e664ad47ddb4ce5a22079f3b3669080" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37681 } } }
I20260929 00:49:50.094837 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:50.095140 13237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:50.099144 13237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:49:50.104377 13237 catalog_manager.cc:1347] Generated new cluster ID: f667fb44771d4349b11055c7dec95791
I20260929 00:49:50.104458 13237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:50.123647 13237 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:50.124704 13237 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:50.134768 13237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Generated new TSK 0
I20260929 00:49:50.135566 13237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:49:50.166689 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37681
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:50.285027 13249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:50.285355 13249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:50.285491 13249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:50.290782 13249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:50.290884 13249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:50.290967 13249 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:49:50.291018 13249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:50.291091 13249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:49:50.297524 13249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:50.298746 13249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:50.299861 13249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:50.310431 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.310452 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.310443 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:50.311261 13249 server_base.cc:1034] running on GCE node
I20260929 00:49:50.311899 13249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:50.312711 13249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:50.313905 13249 hybrid_clock.cc:648] HybridClock initialized: now 1790642990313873 us; error 45 us; skew 500 ppm
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:50.317097 13249 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:49:50.318030 13249 webserver.cc:460] Webserver started at http://127.11.40.193:43113/ using document root <none> and password file <none>
I20260929 00:49:50.318622 13249 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:50.318713 13249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:50.318953 13249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:50.321151 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "a44dee18ddeee3326ff637a82cf03897"
server_key_iv: "5cb285989501feef728709a9e7906fe7"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.321731 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "a44dee18ddeee3326ff637a82cf03897"
server_key_iv: "5cb285989501feef728709a9e7906fe7"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.325759 13249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:49:50.328493 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.329952 13249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:49:50.330111 13249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "a44dee18ddeee3326ff637a82cf03897"
server_key_iv: "5cb285989501feef728709a9e7906fe7"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.330243 13249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:50.345578 13249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:50.349159 13249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:50.349382 13249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:50.350863 13249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:50.350946 13249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.351022 13249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:50.351066 13249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.361991 13249 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:46801
I20260929 00:49:50.362087 13379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:46801 every 8 connection(s)
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:50.363085 13249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:49:50.364902 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13249
I20260929 00:49:50.365036 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:49:50.365353 11427 external_mini_cluster.cc:1428] Setting key 8e67c432f7c4c91845dc1d8206da12bd
I20260929 00:49:50.367753 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.352008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:41087 (local address 127.11.40.254:37681)
0929 00:49:50.352178 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.352184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.353234 (+ 1050us) server_negotiation.cc:407] Connection header received
0929 00:49:50.354602 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.354607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.354670 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.354727 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.356799 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.357824 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.359555 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.360003 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.360124 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.363336 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.363339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.363342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.363377 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.365315 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.365937 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.365966 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.365969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.366066 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.366447 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.366450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.366452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.366662 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.366760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.367386 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.367527 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:49:50.368523 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.352498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.193:41087)
0929 00:49:50.353035 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.353092 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.354309 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.355146 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.355153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.355765 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.356616 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.356655 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.357969 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.357972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.359401 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.359411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.360163 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.360167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.360405 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.361175 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.361193 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.363112 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.365467 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.365480 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.365492 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.365816 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.366181 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.366184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.366186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.366346 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.366838 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.366847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.367140 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.367312 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.367614 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":153,"threads_started":1}
I20260929 00:49:50.370057 13380 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37681
I20260929 00:49:50.370218 13380 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:50.370813 13380 heartbeater.cc:511] Master 127.11.40.254:37681 requested a full tablet report, sending...
I20260929 00:49:50.372617 13176 ts_manager.cc:194] Registered new tserver with Master: 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801)
I20260929 00:49:50.374338 13176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:41087
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:49:50.382542 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.372242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.193:37615)
0929 00:49:50.372505 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.372526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.372597 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.375422 (+ 2825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.375426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.375654 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.375783 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.375792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.376687 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.376690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.377404 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.377414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.377868 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.377871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.377943 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.378503 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.378514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.378938 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.381021 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.381023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.381025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.381247 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.381569 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.381571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.381572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.381651 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.382115 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.382117 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.382193 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.382274 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.382385 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":108,"threads_started":1}
I20260929 00:49:50.382735 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.375032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37615 (local address 127.11.40.254:37681)
0929 00:49:50.375160 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.375165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.375181 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:49:50.375225 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.375230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.375286 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.375398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.375959 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.376548 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.377530 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.377738 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.377821 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.379080 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.379083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.379085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.379106 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.380853 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.381359 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.381365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.381366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.381463 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.381748 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.381754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.381759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.382017 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.382126 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.382361 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.382497 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:49:50.396713 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37681
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:50.511031 13389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:50.511302 13389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:50.511384 13389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:50.515472 13389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:50.515542 13389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:50.515609 13389 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:49:50.515642 13389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:50.515729 13389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:49:50.521221 13389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:50.522328 13389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:50.523372 13389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:50.533020 13395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.532999 13397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.532981 13394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:50.533181 13389 server_base.cc:1034] running on GCE node
I20260929 00:49:50.533825 13389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:50.534515 13389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:50.535691 13389 hybrid_clock.cc:648] HybridClock initialized: now 1790642990535643 us; error 58 us; skew 500 ppm
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:50.538833 13389 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:49:50.539746 13389 webserver.cc:460] Webserver started at http://127.11.40.194:36229/ using document root <none> and password file <none>
I20260929 00:49:50.540300 13389 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:50.540364 13389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:50.540539 13389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:50.542708 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "004fed473707a44a445e6156578b3b92"
server_key_iv: "c3c5ffb604dae0b0c8a6c2739976cf3f"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.543226 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "004fed473707a44a445e6156578b3b92"
server_key_iv: "c3c5ffb604dae0b0c8a6c2739976cf3f"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.547056 13389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:49:50.549767 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.551180 13389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:50.551328 13389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "004fed473707a44a445e6156578b3b92"
server_key_iv: "c3c5ffb604dae0b0c8a6c2739976cf3f"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.551465 13389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:50.566926 13389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:50.570621 13389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:50.570823 13389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:50.572391 13389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:50.572466 13389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.572548 13389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:50.572589 13389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:50.583966 13389 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:34431
I20260929 00:49:50.584174 13519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:34431 every 8 connection(s)
I20260929 00:49:50.585042 13389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:49:50.586735 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.573525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:51401 (local address 127.11.40.254:37681)
0929 00:49:50.573690 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.573695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.574536 (+ 841us) server_negotiation.cc:407] Connection header received
0929 00:49:50.575545 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.575549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.575607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.575713 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.577142 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.578074 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.579384 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.579623 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.579737 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.582816 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.582819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.582822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.582858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.584543 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.585118 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.585122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.585124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.585212 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.585664 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.585667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.585669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.585909 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.586011 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.586512 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.586616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:49:50.587540 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.573888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.194:51401)
0929 00:49:50.574351 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.574402 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.575315 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.575909 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.575916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.576288 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.576954 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.576973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.578210 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.578216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.579262 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.579271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.579787 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.579792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.579963 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.580632 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.580654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.582640 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.584661 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.584667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.584682 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.585012 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.585305 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.585309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.585311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.585545 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.586024 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.586033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.586273 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.586430 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.586692 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":132,"threads_started":1}
I20260929 00:49:50.588688 13520 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37681
I20260929 00:49:50.588846 13520 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:50.589462 13520 heartbeater.cc:511] Master 127.11.40.254:37681 requested a full tablet report, sending...
I20260929 00:49:50.590562 13177 ts_manager.cc:194] Registered new tserver with Master: 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431)
I20260929 00:49:50.591265 13177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:51401
I20260929 00:49:50.593379 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13389
I20260929 00:49:50.593468 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:49:50.593747 11427 external_mini_cluster.cc:1428] Setting key 2a65c76d1d2d8e606e744b7c7da111b8
I20260929 00:49:50.600757 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.591046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.194:43477)
0929 00:49:50.591687 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.591699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.591815 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.592046 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.592049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.592252 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.592357 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.592363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.593198 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.593201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.593901 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.593945 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.594479 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.594482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.594558 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.596724 (+ 2166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.596735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.597119 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.599172 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.599174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.599176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.599367 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.599683 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.599689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.599694 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.599778 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.600206 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.600209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.600304 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.600405 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.600558 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":114,"threads_started":1}
I20260929 00:49:50.600782 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.591182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:43477 (local address 127.11.40.254:37681)
0929 00:49:50.591308 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.591311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.591773 (+ 462us) server_negotiation.cc:407] Connection header received
0929 00:49:50.591895 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.591898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.591941 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.592023 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.592469 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.593059 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.594051 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.594363 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.594480 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.597292 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.597298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.597304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.597333 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.599054 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.599502 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.599505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.599507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.599591 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.599875 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.599878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.599880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.600109 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.600217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.600539 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.600653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:49:50.625388 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37681
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:50.740052 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:50.740337 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:50.740386 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:50.744413 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:50.744485 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:50.744527 13529 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:49:50.744541 13529 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:49:50.744575 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:49:50.750075 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:50.751227 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:50.752264 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:50.762534 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:50.762768 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:50.762573 13529 server_base.cc:1034] running on GCE node
W20260929 00:49:50.762535 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:50.763475 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:50.764156 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:50.765316 13529 hybrid_clock.cc:648] HybridClock initialized: now 1790642990765301 us; error 28 us; skew 500 ppm
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:50.768365 13529 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:49:50.769374 13529 webserver.cc:460] Webserver started at http://127.11.40.195:34985/ using document root <none> and password file <none>
I20260929 00:49:50.769989 13529 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:50.770053 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:50.770265 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:50.772363 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "922044b7663748b3a59ba387ae434486"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "8a9efb33c68512288eb1c59f882c2638"
server_key_iv: "abde4b36dead895de4ed2ffd7e660123"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.773067 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "922044b7663748b3a59ba387ae434486"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "8a9efb33c68512288eb1c59f882c2638"
server_key_iv: "abde4b36dead895de4ed2ffd7e660123"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.777060 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:49:50.779623 13544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.781006 13529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:50.781138 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "922044b7663748b3a59ba387ae434486"
format_stamp: "Formatted at 2026-09-29 00:49:50 on dist-test-slave-5309"
server_key: "8a9efb33c68512288eb1c59f882c2638"
server_key_iv: "abde4b36dead895de4ed2ffd7e660123"
server_key_version: "encryptionkey@0"
I20260929 00:49:50.781256 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:50.805343 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:50.808637 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:50.808867 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:50.810456 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:50.810528 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:50.810598 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:50.810643 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:50.821933 13529 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:45355
I20260929 00:49:50.821955 13659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:45355 every 8 connection(s)
I20260929 00:49:50.823294 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:49:50.826056 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.811437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:56959 (local address 127.11.40.254:37681)
0929 00:49:50.811562 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.811571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.812778 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 00:49:50.814030 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.814034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.814099 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.814180 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.815995 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.816620 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.817913 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.818156 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.818290 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.821889 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.821893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.821896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.821935 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.823734 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.824424 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.824428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.824430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.824536 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.824940 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.824943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.824944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.825141 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.825258 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.825751 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.825884 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:49:50.826831 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.812000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.195:56959)
0929 00:49:50.812562 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.812609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.813800 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.814369 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.814378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.815080 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.815722 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.815738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.816766 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.816770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.817764 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.817774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.818358 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.818363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.818552 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.819928 (+ 1376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.819956 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.821685 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.823899 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.823905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.823921 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.824290 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.824643 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.824646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.824648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.824788 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.825254 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.825260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.825509 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.825690 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.826032 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":142,"threads_started":1}
I20260929 00:49:50.828042 13660 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37681
I20260929 00:49:50.828189 13660 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:50.828783 13660 heartbeater.cc:511] Master 127.11.40.254:37681 requested a full tablet report, sending...
I20260929 00:49:50.829910 13176 ts_manager.cc:194] Registered new tserver with Master: 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355)
I20260929 00:49:50.830578 13176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:56959
I20260929 00:49:50.832544 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13529
I20260929 00:49:50.832664 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:49:50.832993 11427 external_mini_cluster.cc:1428] Setting key a0b4d119ecaf3802a49befb5a2060c12
I20260929 00:49:50.835059 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:49:50.837766 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.830249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:43539 (local address 127.11.40.254:37681)
0929 00:49:50.830857 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.830860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.830871 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:49:50.830898 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.830900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.830929 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.830993 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.831465 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.831987 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.832915 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.833196 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.833297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.834544 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.834548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.834551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.834574 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.836071 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.836536 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.836539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.836541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.836625 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.836976 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.836979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.836981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.837166 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.837249 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.837565 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.837656 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553}
I20260929 00:49:50.837911 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.830196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.11.40.195:43539)
0929 00:49:50.830448 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.830460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.830530 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.831019 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.831022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.831213 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:50.831312 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.831317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.832118 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.832120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.832775 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.832783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.833309 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.833313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.833418 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.833972 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:50.833983 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:50.834427 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:50.836179 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.836185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.836187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.836431 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.836748 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:50.836752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:50.836755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:50.836885 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:50.837261 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:50.837266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:50.837403 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.837527 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.837685 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:49:50 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:50.869087 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.853915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.11.40.254:37681)
0929 00:49:50.854069 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.854074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.854089 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:49:50.854136 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.854138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.854179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.854421 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:50.855375 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.855994 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.856874 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.857093 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.857190 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.865943 (+ 8753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:50.865946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:50.865949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:50.865982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:50.867372 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.867857 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.867860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.867861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.867937 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:50.868228 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:50.868230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:50.868231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:50.868410 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:50.868508 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.868729 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.868892 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:49:50.872397 13176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:49:50.875689 13176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:50.890852 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.884556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:34431 (local address 127.0.0.1:39280)
0929 00:49:50.885491 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.885504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.885591 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.886408 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.886417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.886447 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:50.886618 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.886626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.888166 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.888169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.889417 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.889470 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890528 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.890624 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890644 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.890699 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.890750 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20260929 00:49:50.890939 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.885064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.11.40.194:34431)
0929 00:49:50.885536 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.885556 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.885581 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:49:50.885738 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.885744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.886238 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.886407 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:50.886757 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.888009 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.889600 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890408 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890517 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890669 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.890719 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.890795 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":112,"threads_started":1}
I20260929 00:49:50.890940 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.884484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.11.40.195:45355)
0929 00:49:50.886126 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.886137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.886157 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:49:50.886228 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.886233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.886404 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.886662 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:50.886841 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.887607 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.889259 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890123 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890412 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890601 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.890669 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.890774 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":111,"threads_started":1}
I20260929 00:49:50.891395 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.885452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.11.40.193:46801)
0929 00:49:50.885840 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:50.885849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:50.885907 (+ 58us) server_negotiation.cc:407] Connection header received
0929 00:49:50.886120 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:50.886125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:50.886345 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:50.886499 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:50.886905 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.887996 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.889537 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890101 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890407 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:50.891196 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 00:49:50.891284 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":88,"threads_started":1}
I20260929 00:49:50.890852 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.884433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45355 (local address 127.0.0.1:59552)
0929 00:49:50.885038 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.885097 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.885232 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.886552 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.886555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.886572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:50.886739 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.886746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.887722 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.887729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.889123 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.889135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890273 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.890392 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890443 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.890548 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.890674 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":67,"spinlock_wait_cycles":896,"thread_start_us":99,"threads_started":1}
I20260929 00:49:50.891265 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:50.885262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:46801 (local address 127.0.0.1:56750)
0929 00:49:50.885836 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:50.885849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:50.885946 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:50.886627 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:50.886630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:50.886647 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:50.886776 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.886782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.888122 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.888125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.889406 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:50.889413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:50.890232 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:50.890236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:50.890885 (+ 649us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:50.890969 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 00:49:50.891061 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:49:50.891153 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":62,"thread_start_us":102,"threads_started":1}
I20260929 00:49:50.894490 13594 tablet_service.cc:1467] Processing CreateTablet for tablet d02078f8426e4c15b5fd873260392153 (DEFAULT_TABLE table=test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:50.894600 13440 tablet_service.cc:1467] Processing CreateTablet for tablet d02078f8426e4c15b5fd873260392153 (DEFAULT_TABLE table=test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:50.894587 13314 tablet_service.cc:1467] Processing CreateTablet for tablet d02078f8426e4c15b5fd873260392153 (DEFAULT_TABLE table=test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:50.895546 13594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02078f8426e4c15b5fd873260392153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:50.895570 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02078f8426e4c15b5fd873260392153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:50.895571 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02078f8426e4c15b5fd873260392153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:50.901932 13683 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Bootstrap starting.
I20260929 00:49:50.903590 13684 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Bootstrap starting.
I20260929 00:49:50.903962 13685 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Bootstrap starting.
I20260929 00:49:50.904422 13683 tablet_bootstrap.cc:654] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:50.905392 13683 log.cc:824] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:50.905948 13684 tablet_bootstrap.cc:654] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:50.906391 13685 tablet_bootstrap.cc:654] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:50.906772 13684 log.cc:824] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:50.907166 13685 log.cc:824] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:50.907529 13683 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: No bootstrap required, opened a new log
I20260929 00:49:50.907737 13683 ts_tablet_manager.cc:1397] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:49:50.909250 13684 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: No bootstrap required, opened a new log
I20260929 00:49:50.909250 13685 tablet_bootstrap.cc:492] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: No bootstrap required, opened a new log
I20260929 00:49:50.909435 13684 ts_tablet_manager.cc:1397] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:49:50.909435 13685 ts_tablet_manager.cc:1397] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:49:50.911212 13683 raft_consensus.cc:348] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.911499 13683 raft_consensus.cc:374] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:50.911540 13683 raft_consensus.cc:729] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922044b7663748b3a59ba387ae434486, State: Initialized, Role: FOLLOWER
I20260929 00:49:50.911937 13683 consensus_queue.cc:260] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.912544 13660 heartbeater.cc:503] Master 127.11.40.254:37681 was elected leader, sending a full tablet report...
I20260929 00:49:50.912956 13683 ts_tablet_manager.cc:1428] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260929 00:49:50.913492 13685 raft_consensus.cc:348] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.913757 13685 raft_consensus.cc:374] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:50.913815 13685 raft_consensus.cc:729] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f2ea5b58fe47a88ca6daca372bcb73, State: Initialized, Role: FOLLOWER
I20260929 00:49:50.913982 13684 raft_consensus.cc:348] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.914245 13684 raft_consensus.cc:374] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:50.914306 13684 raft_consensus.cc:729] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Initialized, Role: FOLLOWER
I20260929 00:49:50.914253 13685 consensus_queue.cc:260] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.914718 13684 consensus_queue.cc:260] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:50.915287 13685 ts_tablet_manager.cc:1428] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:49:50.915612 13684 ts_tablet_manager.cc:1428] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:49:50.915741 13520 heartbeater.cc:503] Master 127.11.40.254:37681 was elected leader, sending a full tablet report...
I20260929 00:49:50.915937 13380 heartbeater.cc:503] Master 127.11.40.254:37681 was elected leader, sending a full tablet report...
I20260929 00:49:51.013605 13689 raft_consensus.cc:482] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:51.013878 13689 raft_consensus.cc:504] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:51.015101 13689 leader_election.cc:288] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801)
I20260929 00:49:51.019541 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.015352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:46801 (local address 127.11.40.195:39855)
0929 00:49:51.015572 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.015589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.015692 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.015994 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.015997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.016018 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:51.016161 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.016170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.017171 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.017174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.018203 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.018213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.019041 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.019045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.019169 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.019195 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.019284 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.019364 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":2}
I20260929 00:49:51.019616 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.015467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:39855 (local address 127.11.40.193:46801)
0929 00:49:51.015589 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.015594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.015642 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:49:51.015810 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.015815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.015878 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.015963 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:51.016272 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.017062 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.018331 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.018899 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.018996 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.019037 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.019355 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.019470 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:49:51.019942 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.015468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:50719 (local address 127.11.40.194:34431)
0929 00:49:51.015589 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.015594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.015927 (+ 333us) server_negotiation.cc:407] Connection header received
0929 00:49:51.016052 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.016055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.016102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.016186 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:51.016883 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.017588 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.018756 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.019320 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.019431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.019475 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.019723 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.019807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:49:51.020264 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.015352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:34431 (local address 127.11.40.195:50719)
0929 00:49:51.015809 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.015822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.015954 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.016208 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.016210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.016241 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:51.016695 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.016705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.017711 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.017714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.018628 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.018638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.019451 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.019455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.019555 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.019578 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.020100 (+ 522us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.020160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":53,"thread_start_us":151,"threads_started":1}
I20260929 00:49:51.020453 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02078f8426e4c15b5fd873260392153" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" is_pre_election: true
I20260929 00:49:51.020831 13334 raft_consensus.cc:2454] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:51.020884 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02078f8426e4c15b5fd873260392153" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" is_pre_election: true
I20260929 00:49:51.021258 13474 raft_consensus.cc:2454] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:51.021406 13545 leader_election.cc:302] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:51.021701 13689 raft_consensus.cc:2781] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:51.021793 13689 raft_consensus.cc:482] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:51.021826 13689 raft_consensus.cc:3037] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.022928 13689 raft_consensus.cc:504] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:51.023326 13689 leader_election.cc:288] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Requested vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801)
I20260929 00:49:51.023756 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02078f8426e4c15b5fd873260392153" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
I20260929 00:49:51.023761 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02078f8426e4c15b5fd873260392153" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
I20260929 00:49:51.023869 13334 raft_consensus.cc:3037] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.023893 13474 raft_consensus.cc:3037] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.025009 13334 raft_consensus.cc:2454] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:51.025112 13474 raft_consensus.cc:2454] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:51.025331 13545 leader_election.cc:302] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:51.025524 13689 raft_consensus.cc:2781] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:51.025738 13689 raft_consensus.cc:686] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 1 LEADER]: Becoming Leader. State: Replica: 922044b7663748b3a59ba387ae434486, State: Running, Role: LEADER
I20260929 00:49:51.026145 13689 consensus_queue.cc:237] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } }
I20260929 00:49:51.029693 13175 catalog_manager.cc:5495] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 922044b7663748b3a59ba387ae434486 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } health_report { overall_health: UNKNOWN } } }
W20260929 00:49:51.074857 13661 tablet.cc:2369] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 00:49:51 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642990, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
W20260929 00:49:51.086704 13521 tablet.cc:2369] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:49:51.088618 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.078770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37681 (local address 127.0.0.1:46926)
0929 00:49:51.079111 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.079126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.079268 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.079543 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.079547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.080066 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:51.080195 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.080203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.081018 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.081022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.081940 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.081950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.082380 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.082384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.082481 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.083086 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:51.083100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:51.084717 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:51.086694 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:51.086706 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:51.086713 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:51.087095 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:51.087399 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:51.087405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:51.087409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:51.087513 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:51.087970 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:51.087977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:51.088125 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.088236 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.088387 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260929 00:49:51.088630 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.078815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.11.40.254:37681)
0929 00:49:51.078950 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.078955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.079214 (+ 259us) server_negotiation.cc:407] Connection header received
0929 00:49:51.079373 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.079376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.079420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.079491 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:51.080315 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.080910 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.082060 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.082265 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.082343 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.084915 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:51.084918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:51.084920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:51.084942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:51.086561 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:51.087213 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:51.087215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:51.087217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:51.087302 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:51.087604 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:51.087606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:51.087608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:51.087799 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:51.087896 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.088290 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.088445 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":46}
I20260929 00:49:51.092988 13175 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:46926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:49:51.093791 13175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:51.099078 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 73c904fbcb1747f9ac41496bd3ac335c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7a88c16dc234f449c98da0cb372dae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:51.099057 13314 tablet_service.cc:1467] Processing CreateTablet for tablet 73c904fbcb1747f9ac41496bd3ac335c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7a88c16dc234f449c98da0cb372dae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:51.099326 13594 tablet_service.cc:1467] Processing CreateTablet for tablet 73c904fbcb1747f9ac41496bd3ac335c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7a88c16dc234f449c98da0cb372dae6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:49:51.099795 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c904fbcb1747f9ac41496bd3ac335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.099795 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c904fbcb1747f9ac41496bd3ac335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.099871 13594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c904fbcb1747f9ac41496bd3ac335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.102581 13685 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: Bootstrap starting.
I20260929 00:49:51.102962 13683 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: Bootstrap starting.
I20260929 00:49:51.103478 13684 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: Bootstrap starting.
I20260929 00:49:51.103667 13685 tablet_bootstrap.cc:654] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.103991 13683 tablet_bootstrap.cc:654] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.104493 13684 tablet_bootstrap.cc:654] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.105139 13685 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: No bootstrap required, opened a new log
I20260929 00:49:51.105207 13685 ts_tablet_manager.cc:1397] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:51.105693 13685 raft_consensus.cc:348] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.105823 13685 raft_consensus.cc:374] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.105856 13684 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: No bootstrap required, opened a new log
I20260929 00:49:51.105871 13685 raft_consensus.cc:729] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f2ea5b58fe47a88ca6daca372bcb73, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.105939 13684 ts_tablet_manager.cc:1397] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 00:49:51.106051 13685 consensus_queue.cc:260] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.106115 13683 tablet_bootstrap.cc:492] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: No bootstrap required, opened a new log
I20260929 00:49:51.106199 13683 ts_tablet_manager.cc:1397] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:51.106437 13684 raft_consensus.cc:348] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.106572 13684 raft_consensus.cc:374] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.106606 13684 raft_consensus.cc:729] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.106674 13683 raft_consensus.cc:348] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.106688 13690 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.106741 13684 consensus_queue.cc:260] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.106791 13683 raft_consensus.cc:374] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.106825 13683 raft_consensus.cc:729] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922044b7663748b3a59ba387ae434486, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.106827 13690 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.106958 13683 consensus_queue.cc:260] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.107009 13684 ts_tablet_manager.cc:1428] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:49:51.107007 13691 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.107170 13691 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.107344 13689 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.107462 13689 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.107446 13685 ts_tablet_manager.cc:1428] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:49:51.108227 13683 ts_tablet_manager.cc:1428] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260929 00:49:51.114645 13381 tablet.cc:2369] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:49:51.300087 13689 raft_consensus.cc:482] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:51.300238 13689 raft_consensus.cc:504] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.300711 13689 leader_election.cc:288] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801), 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431)
I20260929 00:49:51.301210 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c904fbcb1747f9ac41496bd3ac335c" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" is_pre_election: true
I20260929 00:49:51.301286 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c904fbcb1747f9ac41496bd3ac335c" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" is_pre_election: true
I20260929 00:49:51.301412 13334 raft_consensus.cc:2454] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:51.301455 13474 raft_consensus.cc:2454] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:51.301755 13545 leader_election.cc:302] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:51.301970 13689 raft_consensus.cc:2781] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:51.302033 13689 raft_consensus.cc:482] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:51.302062 13689 raft_consensus.cc:3037] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.303030 13689 raft_consensus.cc:504] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.303356 13689 leader_election.cc:288] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Requested vote from peers 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801), 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431)
I20260929 00:49:51.303699 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c904fbcb1747f9ac41496bd3ac335c" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
I20260929 00:49:51.303814 13334 raft_consensus.cc:3037] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.303776 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c904fbcb1747f9ac41496bd3ac335c" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
I20260929 00:49:51.303920 13474 raft_consensus.cc:3037] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.304780 13334 raft_consensus.cc:2454] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:51.304965 13474 raft_consensus.cc:2454] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:51.305183 13545 leader_election.cc:302] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:51.305361 13689 raft_consensus.cc:2781] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:51.305456 13689 raft_consensus.cc:686] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [term 1 LEADER]: Becoming Leader. State: Replica: 922044b7663748b3a59ba387ae434486, State: Running, Role: LEADER
I20260929 00:49:51.305615 13689 consensus_queue.cc:237] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.306298 13693 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: TxnStatusTablet state changed. Reason: New leader 922044b7663748b3a59ba387ae434486. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.306414 13693 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:51.306718 13705 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:49:51.306962 13705 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:49:51.307444 13176 catalog_manager.cc:5495] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 922044b7663748b3a59ba387ae434486 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } health_report { overall_health: HEALTHY } } }
I20260929 00:49:51.349191 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.345275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45355 (local address 127.0.0.1:59558)
0929 00:49:51.345485 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.345503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.345645 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.345984 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.345987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.345994 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:51.346104 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.346110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.346837 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.346841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.347502 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.347510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.347989 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.347992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.348087 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.348131 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.348860 (+ 729us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.348876 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.348966 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.349066 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:49:51.349191 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.345370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.11.40.195:45355)
0929 00:49:51.345487 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.345492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.345573 (+ 81us) server_negotiation.cc:407] Connection header received
0929 00:49:51.345792 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.345795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.345852 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.345971 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.346206 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.346724 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.347627 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.347884 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.347984 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.348291 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.348722 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.348873 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.348978 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.349037 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:49:51.354782 13474 raft_consensus.cc:1261] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.354791 13334 raft_consensus.cc:1261] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.355526 13693 consensus_queue.cc:1035] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.355979 13693 consensus_queue.cc:1035] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.360069 13690 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: TxnStatusTablet state changed. Reason: New leader 922044b7663748b3a59ba387ae434486. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.360227 13690 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.360252 13691 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: TxnStatusTablet state changed. Reason: New leader 922044b7663748b3a59ba387ae434486. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.360417 13691 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.364790 13693 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.364943 13693 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:51.366576 13691 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.366689 13691 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 3c296035b5704f9f92c65a5fcb0e4d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.367354 13710 mvcc.cc:205] Tried to move back new op lower bound from 7334473692582080512 to 7334473692389257216. Current Snapshot: MvccSnapshot[applied={T|T < 7334473692582080512}]
I20260929 00:49:51.373922 13693 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.374060 13693 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 922044b7663748b3a59ba387ae434486: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:49:51.376644 13709 mvcc.cc:205] Tried to move back new op lower bound from 7334473692582080512 to 7334473692389257216. Current Snapshot: MvccSnapshot[applied={T|T < 7334473692582080512}]
I20260929 00:49:51.376864 13690 tablet_replica.cc:439] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } } }
I20260929 00:49:51.376978 13690 tablet_replica.cc:442] T 73c904fbcb1747f9ac41496bd3ac335c P 80f2ea5b58fe47a88ca6daca372bcb73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:49:51.379545 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.371947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.11.40.195:45355)
0929 00:49:51.372063 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.372067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.373539 (+ 1472us) server_negotiation.cc:407] Connection header received
0929 00:49:51.373736 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.373739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.373788 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.373835 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.374408 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.375197 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.377422 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.377790 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.377869 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.379052 (+ 1183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.379139 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.379211 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.379343 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.379406 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:49:51.391070 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.387216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:35437 (local address 127.11.40.195:45355)
0929 00:49:51.387340 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.387344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.387540 (+ 196us) server_negotiation.cc:407] Connection header received
0929 00:49:51.387665 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.387667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.387707 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.387762 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.388044 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.388663 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.390007 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.390223 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.390325 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.390664 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.390731 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.390785 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.390907 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.390950 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":50}
I20260929 00:49:51.391136 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.387155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45355 (local address 127.11.40.195:35437)
0929 00:49:51.387453 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.387468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.387575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.387787 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.387790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.387798 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:51.387943 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.387949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.388761 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.388764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.389879 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.389890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.390331 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.390335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.390461 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.390515 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.390809 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.390815 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.390880 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.390959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":20,"thread_start_us":117,"threads_started":1}
I20260929 00:49:51.398012 13334 raft_consensus.cc:1261] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.398061 13474 raft_consensus.cc:1261] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.398468 13721 consensus_queue.cc:1035] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.398625 13693 consensus_queue.cc:1035] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.402645 13708 mvcc.cc:205] Tried to move back new op lower bound from 7334473692762230784 to 7334473691246170112. Current Snapshot: MvccSnapshot[applied={T|T < 7334473692762230784}]
I20260929 00:49:51.440436 13176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:51.440613 13176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:51.443528 13176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Sending DeleteTablet for 3 replicas of tablet d02078f8426e4c15b5fd873260392153
I20260929 00:49:51.444520 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet d02078f8426e4c15b5fd873260392153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:51 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39280
I20260929 00:49:51.444520 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet d02078f8426e4c15b5fd873260392153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:51 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:56750
I20260929 00:49:51.445112 13176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:49:51.445353 13726 tablet_replica.cc:330] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: stopping tablet replica
I20260929 00:49:51.445305 13593 tablet_service.cc:1514] Processing DeleteTablet for tablet d02078f8426e4c15b5fd873260392153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:51 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59552
W20260929 00:49:51.445568 13176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:51.445736 13726 raft_consensus.cc:2229] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:51.446007 13726 raft_consensus.cc:2258] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:51.446199 13728 tablet_replica.cc:330] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: stopping tablet replica
I20260929 00:49:51.446480 13727 tablet_replica.cc:330] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: stopping tablet replica
I20260929 00:49:51.446828 13727 raft_consensus.cc:2229] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:51.447052 13727 raft_consensus.cc:2258] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:51.447827 13726 ts_tablet_manager.cc:1905] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:51.448463 13727 ts_tablet_manager.cc:1905] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:51.451412 13726 ts_tablet_manager.cc:1918] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:51.451398 13440 tablet_service.cc:1467] Processing CreateTablet for tablet d94b089d5b22432f90681edb71408652 (DEFAULT_TABLE table=test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:51.451682 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b089d5b22432f90681edb71408652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.452031 13728 raft_consensus.cc:2229] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:51.452528 13728 raft_consensus.cc:2258] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:51.452910 13726 log.cc:1196] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/d02078f8426e4c15b5fd873260392153
I20260929 00:49:51.452972 13593 tablet_service.cc:1467] Processing CreateTablet for tablet d94b089d5b22432f90681edb71408652 (DEFAULT_TABLE table=test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:51.453274 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b089d5b22432f90681edb71408652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.453506 13726 ts_tablet_manager.cc:1939] T d02078f8426e4c15b5fd873260392153 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting consensus metadata
I20260929 00:49:51.451710 13727 ts_tablet_manager.cc:1918] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:51.454489 13727 log.cc:1196] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/d02078f8426e4c15b5fd873260392153
I20260929 00:49:51.454842 13727 ts_tablet_manager.cc:1939] T d02078f8426e4c15b5fd873260392153 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting consensus metadata
I20260929 00:49:51.456024 13683 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Bootstrap starting.
I20260929 00:49:51.456331 13685 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Bootstrap starting.
I20260929 00:49:51.452286 13314 tablet_service.cc:1467] Processing CreateTablet for tablet d94b089d5b22432f90681edb71408652 (DEFAULT_TABLE table=test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:51.457165 13683 tablet_bootstrap.cc:654] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.457309 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b089d5b22432f90681edb71408652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:51.457401 13685 tablet_bootstrap.cc:654] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.458091 13163 catalog_manager.cc:4841] TS 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431): tablet d02078f8426e4c15b5fd873260392153 (table test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]) successfully deleted
I20260929 00:49:51.458663 13685 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: No bootstrap required, opened a new log
I20260929 00:49:51.458756 13685 ts_tablet_manager.cc:1397] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:49:51.459059 13683 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: No bootstrap required, opened a new log
I20260929 00:49:51.459136 13683 ts_tablet_manager.cc:1397] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:51.459241 13685 raft_consensus.cc:348] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.459376 13685 raft_consensus.cc:374] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.459414 13685 raft_consensus.cc:729] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f2ea5b58fe47a88ca6daca372bcb73, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.459544 13685 consensus_queue.cc:260] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.459610 13683 raft_consensus.cc:348] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.459710 13683 raft_consensus.cc:374] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.459743 13683 raft_consensus.cc:729] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922044b7663748b3a59ba387ae434486, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.459874 13685 ts_tablet_manager.cc:1428] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 00:49:51.459815 13161 catalog_manager.cc:4841] TS 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801): tablet d02078f8426e4c15b5fd873260392153 (table test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]) successfully deleted
I20260929 00:49:51.459877 13683 consensus_queue.cc:260] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.460151 13683 ts_tablet_manager.cc:1428] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:49:51.460433 13684 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Bootstrap starting.
I20260929 00:49:51.461501 13684 tablet_bootstrap.cc:654] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:51.462760 13684 tablet_bootstrap.cc:492] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: No bootstrap required, opened a new log
I20260929 00:49:51.462874 13684 ts_tablet_manager.cc:1397] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:49:51.463341 13684 raft_consensus.cc:348] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.463472 13684 raft_consensus.cc:374] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:51.463514 13684 raft_consensus.cc:729] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Initialized, Role: FOLLOWER
I20260929 00:49:51.463656 13684 consensus_queue.cc:260] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.463709 13728 ts_tablet_manager.cc:1905] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:51.463973 13684 ts_tablet_manager.cc:1428] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:49:51.467281 13728 ts_tablet_manager.cc:1918] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:51.467401 13728 log.cc:1196] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/d02078f8426e4c15b5fd873260392153
I20260929 00:49:51.467732 13728 ts_tablet_manager.cc:1939] T d02078f8426e4c15b5fd873260392153 P 922044b7663748b3a59ba387ae434486: Deleting consensus metadata
I20260929 00:49:51.468932 13161 catalog_manager.cc:4841] TS 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355): tablet d02078f8426e4c15b5fd873260392153 (table test-table [id=2d278d0a8905436a86a1e0a92b55aa6d]) successfully deleted
I20260929 00:49:51.505609 13691 raft_consensus.cc:482] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:51.505748 13691 raft_consensus.cc:504] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.506978 13691 leader_election.cc:288] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355)
I20260929 00:49:51.511967 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.507382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:39423 (local address 127.11.40.195:45355)
0929 00:49:51.507511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.507516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.507720 (+ 204us) server_negotiation.cc:407] Connection header received
0929 00:49:51.507967 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.507971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.508025 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.508092 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:51.508462 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.509037 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.510915 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.511310 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.511447 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.511504 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.511733 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.511824 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:49:51.512084 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.507148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:34431 (local address 127.11.40.193:54173)
0929 00:49:51.507551 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.507566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.507678 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.508019 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.508022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.508039 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:51.508164 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.508171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.509267 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.509271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.510756 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.510766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.511635 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.511641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.511752 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.511770 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.511837 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.511925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":122,"thread_start_us":148,"threads_started":1}
I20260929 00:49:51.512084 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.507629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:54173 (local address 127.11.40.194:34431)
0929 00:49:51.507761 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.507768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.507784 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:49:51.507840 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.507843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.507892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.507995 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:49:51.508322 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.509138 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.510900 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.511521 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.511646 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.511697 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.511873 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.511945 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:49:51.512728 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b089d5b22432f90681edb71408652" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" is_pre_election: true
I20260929 00:49:51.512943 13473 raft_consensus.cc:2454] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 0.
I20260929 00:49:51.512981 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.507253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45355 (local address 127.11.40.193:39423)
0929 00:49:51.507632 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.507646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.507791 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.508137 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.508140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.508158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:49:51.508316 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.508325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.509150 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.509156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.510756 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.510766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.511456 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.511460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.511557 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.511610 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.512721 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.512823 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":150,"thread_start_us":90,"threads_started":1}
I20260929 00:49:51.513382 13267 leader_election.cc:302] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 80f2ea5b58fe47a88ca6daca372bcb73; no voters:
I20260929 00:49:51.513487 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b089d5b22432f90681edb71408652" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922044b7663748b3a59ba387ae434486" is_pre_election: true
I20260929 00:49:51.513721 13614 raft_consensus.cc:2454] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 0.
I20260929 00:49:51.513892 13691 raft_consensus.cc:2781] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:51.513952 13691 raft_consensus.cc:482] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:51.513998 13691 raft_consensus.cc:3037] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.515425 13691 raft_consensus.cc:504] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.515825 13691 leader_election.cc:288] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 election: Requested vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355)
I20260929 00:49:51.516201 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b089d5b22432f90681edb71408652" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922044b7663748b3a59ba387ae434486"
I20260929 00:49:51.516223 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b089d5b22432f90681edb71408652" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
I20260929 00:49:51.516322 13614 raft_consensus.cc:3037] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.516351 13473 raft_consensus.cc:3037] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:51.517424 13473 raft_consensus.cc:2454] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 1.
I20260929 00:49:51.517530 13614 raft_consensus.cc:2454] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 1.
I20260929 00:49:51.517751 13267 leader_election.cc:302] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 80f2ea5b58fe47a88ca6daca372bcb73; no voters:
I20260929 00:49:51.517970 13691 raft_consensus.cc:2781] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:51.518251 13691 raft_consensus.cc:686] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 LEADER]: Becoming Leader. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Running, Role: LEADER
I20260929 00:49:51.518519 13691 consensus_queue.cc:237] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:51.521054 13176 catalog_manager.cc:5495] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20260929 00:49:51.543443 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.539381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.11.40.193:46801)
0929 00:49:51.539631 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.539639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.539657 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:49:51.539756 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.539758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.539815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.539963 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.540219 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.540859 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.541823 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.542089 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.542205 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.542499 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.542992 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.543139 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.543185 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.543245 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260929 00:49:51.555406 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.551429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:52179 (local address 127.11.40.195:45355)
0929 00:49:51.551575 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.551579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.551786 (+ 207us) server_negotiation.cc:407] Connection header received
0929 00:49:51.551910 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.551913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.552010 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.552091 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.552390 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.552998 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.554241 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.554461 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.554671 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.554996 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.555074 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.555126 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.555228 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.555287 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:49:51.555908 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.551282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45355 (local address 127.11.40.193:52179)
0929 00:49:51.551674 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.551689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.551807 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.552119 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.552123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.552130 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:51.552279 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.552285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.553099 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.553103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.554076 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.554088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.554685 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.554689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.554794 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.554864 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.555147 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.555154 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.555217 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.555332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":138,"threads_started":1}
I20260929 00:49:51.562286 13614 raft_consensus.cc:1261] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Refusing update from remote peer 3c296035b5704f9f92c65a5fcb0e4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.562883 13473 raft_consensus.cc:1261] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Refusing update from remote peer 3c296035b5704f9f92c65a5fcb0e4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:51.562924 13734 consensus_queue.cc:1035] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.563301 13691 consensus_queue.cc:1035] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:51.586002 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.582044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:46801 (local address 127.11.40.195:36671)
0929 00:49:51.582189 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:51.582205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:51.582289 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:51.582691 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:51.582694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:51.582703 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:49:51.582811 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.582817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.583766 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.583769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.584603 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:51.584614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.585223 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.585227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:51.585309 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.585343 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.585679 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.585685 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:49:51.585744 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 00:49:51.585833 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 00:49:51.586659 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:51.582257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:36671 (local address 127.11.40.193:46801)
0929 00:49:51.582465 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:51.582472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:51.582487 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:49:51.582531 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:51.582537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:51.582587 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:51.582659 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:51.582911 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.583641 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.584729 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:51.585113 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:51.585216 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:51.585489 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:51.585577 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:51.585640 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:51.586485 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 00:49:51.586543 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260929 00:49:51.587471 13710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:49:53.155198 13176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:53.155387 13176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:53.158376 13176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Sending DeleteTablet for 3 replicas of tablet d94b089d5b22432f90681edb71408652
I20260929 00:49:53.159067 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet d94b089d5b22432f90681edb71408652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39280
I20260929 00:49:53.159083 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet d94b089d5b22432f90681edb71408652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:56750
I20260929 00:49:53.159453 13766 tablet_replica.cc:330] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: stopping tablet replica
I20260929 00:49:53.159610 13766 raft_consensus.cc:2229] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:53.159736 13766 raft_consensus.cc:2258] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:53.160478 13766 ts_tablet_manager.cc:1905] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.160650 13593 tablet_service.cc:1514] Processing DeleteTablet for tablet d94b089d5b22432f90681edb71408652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59552
I20260929 00:49:53.161084 13765 tablet_replica.cc:330] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: stopping tablet replica
I20260929 00:49:53.161207 13765 raft_consensus.cc:2229] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:53.161530 13765 raft_consensus.cc:2258] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:53.162245 13765 ts_tablet_manager.cc:1905] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.163962 13766 ts_tablet_manager.cc:1918] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:49:53.164067 13766 log.cc:1196] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/d94b089d5b22432f90681edb71408652
I20260929 00:49:53.164352 13766 ts_tablet_manager.cc:1939] T d94b089d5b22432f90681edb71408652 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting consensus metadata
I20260929 00:49:53.164925 13765 ts_tablet_manager.cc:1918] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:49:53.165015 13765 log.cc:1196] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/d94b089d5b22432f90681edb71408652
I20260929 00:49:53.165213 13163 catalog_manager.cc:4841] TS 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431): tablet d94b089d5b22432f90681edb71408652 (table test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]) successfully deleted
I20260929 00:49:53.165285 13765 ts_tablet_manager.cc:1939] T d94b089d5b22432f90681edb71408652 P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting consensus metadata
I20260929 00:49:53.165594 13767 tablet_replica.cc:330] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: stopping tablet replica
I20260929 00:49:53.165745 13767 raft_consensus.cc:2229] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:53.165884 13767 raft_consensus.cc:2258] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:53.166034 13161 catalog_manager.cc:4841] TS 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801): tablet d94b089d5b22432f90681edb71408652 (table test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]) successfully deleted
I20260929 00:49:53.166590 13767 ts_tablet_manager.cc:1905] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.168941 13767 ts_tablet_manager.cc:1918] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:49:53.169008 13767 log.cc:1196] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/d94b089d5b22432f90681edb71408652
I20260929 00:49:53.169267 13767 ts_tablet_manager.cc:1939] T d94b089d5b22432f90681edb71408652 P 922044b7663748b3a59ba387ae434486: Deleting consensus metadata
I20260929 00:49:53.169942 13161 catalog_manager.cc:4841] TS 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355): tablet d94b089d5b22432f90681edb71408652 (table test-table [id=eeaf1a5479674b9f88890f6ddfaa3745]) successfully deleted
Sep 29 00:49:53 dist-test-slave-5309 krb5kdc[13132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:49:53 dist-test-slave-5309 krb5kdc[13132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:53.189615 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:53.180169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.11.40.254:37681)
0929 00:49:53.180460 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:53.180471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:53.180556 (+ 85us) server_negotiation.cc:407] Connection header received
0929 00:49:53.180726 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:53.180729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:53.180786 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:53.180917 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:53.181628 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.182212 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.183078 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.183319 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.183432 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:53.186248 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:53.186252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:53.186254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:53.186279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:53.187856 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:53.188304 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:53.188307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:53.188308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:53.188388 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:53.188633 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:53.188635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:53.188636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:53.188865 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:53.188978 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:53.189268 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 00:49:53.189422 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
I20260929 00:49:53.192245 13176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:49:53.192649 13176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:53.197847 13314 tablet_service.cc:1467] Processing CreateTablet for tablet 28240f186dc747c19e3a00bc31b211be (DEFAULT_TABLE table=test-table [id=a514fa794ff147cbba1c27a912070f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.197858 13593 tablet_service.cc:1467] Processing CreateTablet for tablet 28240f186dc747c19e3a00bc31b211be (DEFAULT_TABLE table=test-table [id=a514fa794ff147cbba1c27a912070f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.198184 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28240f186dc747c19e3a00bc31b211be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.198186 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28240f186dc747c19e3a00bc31b211be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.200214 13776 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Bootstrap starting.
I20260929 00:49:53.200423 13777 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Bootstrap starting.
I20260929 00:49:53.200450 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 28240f186dc747c19e3a00bc31b211be (DEFAULT_TABLE table=test-table [id=a514fa794ff147cbba1c27a912070f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.200724 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28240f186dc747c19e3a00bc31b211be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.201188 13776 tablet_bootstrap.cc:654] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.201366 13777 tablet_bootstrap.cc:654] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.202394 13776 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: No bootstrap required, opened a new log
I20260929 00:49:53.202492 13776 ts_tablet_manager.cc:1397] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:49:53.202562 13777 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: No bootstrap required, opened a new log
I20260929 00:49:53.202642 13777 ts_tablet_manager.cc:1397] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:49:53.202950 13776 raft_consensus.cc:348] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.203022 13777 raft_consensus.cc:348] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.203081 13776 raft_consensus.cc:374] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.203115 13776 raft_consensus.cc:729] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.203230 13776 consensus_queue.cc:260] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.203709 13777 raft_consensus.cc:374] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.203773 13777 raft_consensus.cc:729] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922044b7663748b3a59ba387ae434486, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.203900 13780 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Bootstrap starting.
I20260929 00:49:53.203914 13777 consensus_queue.cc:260] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.204223 13777 ts_tablet_manager.cc:1428] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:49:53.204906 13780 tablet_bootstrap.cc:654] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.204995 13776 ts_tablet_manager.cc:1428] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:49:53.206367 13780 tablet_bootstrap.cc:492] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: No bootstrap required, opened a new log
I20260929 00:49:53.206652 13780 ts_tablet_manager.cc:1397] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 00:49:53.207170 13780 raft_consensus.cc:348] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.207301 13780 raft_consensus.cc:374] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.207350 13780 raft_consensus.cc:729] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f2ea5b58fe47a88ca6daca372bcb73, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.207481 13780 consensus_queue.cc:260] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.207823 13780 ts_tablet_manager.cc:1428] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 00:49:53.266124 13742 raft_consensus.cc:482] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:53.266273 13742 raft_consensus.cc:504] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.266707 13742 leader_election.cc:288] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355)
I20260929 00:49:53.267217 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28240f186dc747c19e3a00bc31b211be" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922044b7663748b3a59ba387ae434486" is_pre_election: true
I20260929 00:49:53.267218 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28240f186dc747c19e3a00bc31b211be" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" is_pre_election: true
I20260929 00:49:53.267379 13614 raft_consensus.cc:2454] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 0.
I20260929 00:49:53.267379 13473 raft_consensus.cc:2454] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 0.
I20260929 00:49:53.267736 13265 leader_election.cc:302] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:53.267939 13742 raft_consensus.cc:2781] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:53.268039 13742 raft_consensus.cc:482] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:53.268093 13742 raft_consensus.cc:3037] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.269230 13742 raft_consensus.cc:504] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.269569 13742 leader_election.cc:288] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 election: Requested vote from peers 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431), 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355)
I20260929 00:49:53.269960 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28240f186dc747c19e3a00bc31b211be" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922044b7663748b3a59ba387ae434486"
I20260929 00:49:53.269982 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28240f186dc747c19e3a00bc31b211be" candidate_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
I20260929 00:49:53.270072 13614 raft_consensus.cc:3037] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.270125 13473 raft_consensus.cc:3037] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.271159 13473 raft_consensus.cc:2454] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 1.
I20260929 00:49:53.271184 13614 raft_consensus.cc:2454] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c296035b5704f9f92c65a5fcb0e4d36 in term 1.
I20260929 00:49:53.271499 13265 leader_election.cc:302] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296035b5704f9f92c65a5fcb0e4d36, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:53.271670 13742 raft_consensus.cc:2781] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:53.271768 13742 raft_consensus.cc:686] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 LEADER]: Becoming Leader. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Running, Role: LEADER
I20260929 00:49:53.271909 13742 consensus_queue.cc:237] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.273656 13175 catalog_manager.cc:5495] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20260929 00:49:53.290324 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:53.286913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.11.40.193:46801)
0929 00:49:53.287180 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:53.287188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:53.287200 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:49:53.287239 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:53.287241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:53.287283 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:53.287361 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:53.287639 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.288214 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.289237 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.289477 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.289568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:53.289822 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:53.289896 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:53.289959 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:53.290080 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 00:49:53.290126 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20260929 00:49:53.296991 13473 raft_consensus.cc:1261] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Refusing update from remote peer 3c296035b5704f9f92c65a5fcb0e4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:53.297001 13614 raft_consensus.cc:1261] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Refusing update from remote peer 3c296035b5704f9f92c65a5fcb0e4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:53.297394 13757 consensus_queue.cc:1035] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:53.297554 13742 consensus_queue.cc:1035] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:53.328444 13177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:53.328608 13177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:53.330673 13177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Sending DeleteTablet for 3 replicas of tablet 28240f186dc747c19e3a00bc31b211be
I20260929 00:49:53.331278 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet 28240f186dc747c19e3a00bc31b211be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:56750
I20260929 00:49:53.331293 13593 tablet_service.cc:1514] Processing DeleteTablet for tablet 28240f186dc747c19e3a00bc31b211be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59552
I20260929 00:49:53.331399 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet 28240f186dc747c19e3a00bc31b211be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:53 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39280
I20260929 00:49:53.331478 13767 tablet_replica.cc:330] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: stopping tablet replica
I20260929 00:49:53.331604 13767 raft_consensus.cc:2229] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:53.331622 13765 tablet_replica.cc:330] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: stopping tablet replica
I20260929 00:49:53.331732 13767 raft_consensus.cc:2258] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:53.331746 13765 raft_consensus.cc:2229] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:53.331931 13766 tablet_replica.cc:330] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: stopping tablet replica
I20260929 00:49:53.332000 13765 raft_consensus.cc:2258] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:53.332083 13766 raft_consensus.cc:2229] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:53.332108 13177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:49:53.332227 13766 raft_consensus.cc:2258] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 00:49:53.332561 13177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:49:53.332739 13765 ts_tablet_manager.cc:1905] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.333006 13766 ts_tablet_manager.cc:1905] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.333007 13767 ts_tablet_manager.cc:1905] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:53.335083 13765 ts_tablet_manager.cc:1918] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:53.335136 13765 log.cc:1196] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/28240f186dc747c19e3a00bc31b211be
I20260929 00:49:53.335321 13765 ts_tablet_manager.cc:1939] T 28240f186dc747c19e3a00bc31b211be P 3c296035b5704f9f92c65a5fcb0e4d36: Deleting consensus metadata
I20260929 00:49:53.335455 13767 ts_tablet_manager.cc:1918] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:53.335454 13766 ts_tablet_manager.cc:1918] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:49:53.335523 13767 log.cc:1196] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/28240f186dc747c19e3a00bc31b211be
I20260929 00:49:53.335534 13766 log.cc:1196] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/28240f186dc747c19e3a00bc31b211be
I20260929 00:49:53.335779 13767 ts_tablet_manager.cc:1939] T 28240f186dc747c19e3a00bc31b211be P 922044b7663748b3a59ba387ae434486: Deleting consensus metadata
I20260929 00:49:53.335779 13766 ts_tablet_manager.cc:1939] T 28240f186dc747c19e3a00bc31b211be P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting consensus metadata
I20260929 00:49:53.336527 13161 catalog_manager.cc:4841] TS 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801): tablet 28240f186dc747c19e3a00bc31b211be (table test-table [id=a514fa794ff147cbba1c27a912070f25]) successfully deleted
I20260929 00:49:53.336833 13163 catalog_manager.cc:4841] TS 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431): tablet 28240f186dc747c19e3a00bc31b211be (table test-table [id=a514fa794ff147cbba1c27a912070f25]) successfully deleted
I20260929 00:49:53.337237 13161 catalog_manager.cc:4841] TS 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355): tablet 28240f186dc747c19e3a00bc31b211be (table test-table [id=a514fa794ff147cbba1c27a912070f25]) successfully deleted
I20260929 00:49:53.339520 13314 tablet_service.cc:1467] Processing CreateTablet for tablet 06d24f907a114b45b287d89694fb4383 (DEFAULT_TABLE table=test-table [id=e89800b73c754a75b71d3eeab7664026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.339607 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 06d24f907a114b45b287d89694fb4383 (DEFAULT_TABLE table=test-table [id=e89800b73c754a75b71d3eeab7664026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.339792 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d24f907a114b45b287d89694fb4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.339862 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d24f907a114b45b287d89694fb4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.339897 13593 tablet_service.cc:1467] Processing CreateTablet for tablet 06d24f907a114b45b287d89694fb4383 (DEFAULT_TABLE table=test-table [id=e89800b73c754a75b71d3eeab7664026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:49:53.340116 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d24f907a114b45b287d89694fb4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:53.341646 13780 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Bootstrap starting.
I20260929 00:49:53.342111 13776 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36: Bootstrap starting.
I20260929 00:49:53.342535 13780 tablet_bootstrap.cc:654] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.342676 13777 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Bootstrap starting.
I20260929 00:49:53.342975 13776 tablet_bootstrap.cc:654] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.343509 13777 tablet_bootstrap.cc:654] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:53.343839 13780 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: No bootstrap required, opened a new log
I20260929 00:49:53.343940 13780 ts_tablet_manager.cc:1397] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:49:53.344457 13780 raft_consensus.cc:348] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.344583 13777 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: No bootstrap required, opened a new log
I20260929 00:49:53.344597 13780 raft_consensus.cc:374] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.344636 13780 raft_consensus.cc:729] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f2ea5b58fe47a88ca6daca372bcb73, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.344671 13777 ts_tablet_manager.cc:1397] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:49:53.344833 13776 tablet_bootstrap.cc:492] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36: No bootstrap required, opened a new log
I20260929 00:49:53.344789 13780 consensus_queue.cc:260] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.344919 13776 ts_tablet_manager.cc:1397] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:53.345088 13780 ts_tablet_manager.cc:1428] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 00:49:53.345109 13777 raft_consensus.cc:348] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.345216 13777 raft_consensus.cc:374] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.345248 13777 raft_consensus.cc:729] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922044b7663748b3a59ba387ae434486, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.345311 13776 raft_consensus.cc:348] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.345415 13776 raft_consensus.cc:374] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:53.345378 13777 consensus_queue.cc:260] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.345446 13776 raft_consensus.cc:729] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296035b5704f9f92c65a5fcb0e4d36, State: Initialized, Role: FOLLOWER
I20260929 00:49:53.345577 13776 consensus_queue.cc:260] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.345638 13777 ts_tablet_manager.cc:1428] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:49:53.345839 13776 ts_tablet_manager.cc:1428] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:49:53.424356 13744 raft_consensus.cc:482] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:49:53.424521 13744 raft_consensus.cc:504] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.425041 13744 leader_election.cc:288] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801), 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431)
I20260929 00:49:53.425457 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d24f907a114b45b287d89694fb4383" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" is_pre_election: true
I20260929 00:49:53.425462 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d24f907a114b45b287d89694fb4383" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" is_pre_election: true
I20260929 00:49:53.425623 13473 raft_consensus.cc:2454] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:53.425622 13334 raft_consensus.cc:2454] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 0.
I20260929 00:49:53.425974 13547 leader_election.cc:302] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f2ea5b58fe47a88ca6daca372bcb73, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:53.426144 13744 raft_consensus.cc:2781] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:49:53.426203 13744 raft_consensus.cc:482] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:49:53.426247 13744 raft_consensus.cc:3037] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.427304 13744 raft_consensus.cc:504] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.427711 13744 leader_election.cc:288] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Requested vote from peers 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801), 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431)
I20260929 00:49:53.428069 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d24f907a114b45b287d89694fb4383" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f2ea5b58fe47a88ca6daca372bcb73"
I20260929 00:49:53.428121 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d24f907a114b45b287d89694fb4383" candidate_uuid: "922044b7663748b3a59ba387ae434486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296035b5704f9f92c65a5fcb0e4d36"
I20260929 00:49:53.428232 13334 raft_consensus.cc:3037] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.428232 13473 raft_consensus.cc:3037] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:53.429461 13334 raft_consensus.cc:2454] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:53.429466 13473 raft_consensus.cc:2454] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922044b7663748b3a59ba387ae434486 in term 1.
I20260929 00:49:53.429795 13547 leader_election.cc:302] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f2ea5b58fe47a88ca6daca372bcb73, 922044b7663748b3a59ba387ae434486; no voters:
I20260929 00:49:53.429965 13744 raft_consensus.cc:2781] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:53.430088 13744 raft_consensus.cc:686] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 1 LEADER]: Becoming Leader. State: Replica: 922044b7663748b3a59ba387ae434486, State: Running, Role: LEADER
I20260929 00:49:53.430240 13744 consensus_queue.cc:237] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } }
I20260929 00:49:53.431892 13176 catalog_manager.cc:5495] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 922044b7663748b3a59ba387ae434486 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "922044b7663748b3a59ba387ae434486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922044b7663748b3a59ba387ae434486" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45355 } health_report { overall_health: HEALTHY } } }
I20260929 00:49:53.453826 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:53.450645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.11.40.195:45355)
0929 00:49:53.450935 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:53.450945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:53.450956 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:49:53.450998 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:53.451001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:53.451055 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:53.451144 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:49:53.451383 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.451917 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.452743 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:53.453003 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:53.453118 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:53.453358 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:49:53.453440 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:49:53.453518 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:53.453613 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 00:49:53.453665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260929 00:49:53.460175 13473 raft_consensus.cc:1261] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:53.460304 13334 raft_consensus.cc:1261] T 06d24f907a114b45b287d89694fb4383 P 3c296035b5704f9f92c65a5fcb0e4d36 [term 1 FOLLOWER]: Refusing update from remote peer 922044b7663748b3a59ba387ae434486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:49:53.460578 13744 consensus_queue.cc:1035] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f2ea5b58fe47a88ca6daca372bcb73" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:49:53.460724 13752 consensus_queue.cc:1035] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c296035b5704f9f92c65a5fcb0e4d36" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:49:53.474455 13745 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:49:55.030350 13175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:55.030571 13175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:49:55.033391 13175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e664ad47ddb4ce5a22079f3b3669080: Sending DeleteTablet for 3 replicas of tablet 06d24f907a114b45b287d89694fb4383
I20260929 00:49:55.034076 13594 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d24f907a114b45b287d89694fb4383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:55 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59552
I20260929 00:49:55.034215 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d24f907a114b45b287d89694fb4383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:55 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39280
I20260929 00:49:55.034433 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d24f907a114b45b287d89694fb4383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:49:55 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:56750
I20260929 00:49:55.034651 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13249
I20260929 00:49:55.034672 13819 tablet_replica.cc:330] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: stopping tablet replica
I20260929 00:49:55.035584 13819 raft_consensus.cc:2229] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:49:55.035806 13819 raft_consensus.cc:2258] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:55.036607 13819 ts_tablet_manager.cc:1905] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:55.037050 13818 tablet_replica.cc:330] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: stopping tablet replica
I20260929 00:49:55.037218 13818 raft_consensus.cc:2229] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:49:55.037461 13818 raft_consensus.cc:2258] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:49:55.038262 13818 ts_tablet_manager.cc:1905] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:49:55.039342 13819 ts_tablet_manager.cc:1918] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:49:55.039429 13819 log.cc:1196] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/06d24f907a114b45b287d89694fb4383
I20260929 00:49:55.039671 13819 ts_tablet_manager.cc:1939] T 06d24f907a114b45b287d89694fb4383 P 80f2ea5b58fe47a88ca6daca372bcb73: Deleting consensus metadata
I20260929 00:49:55.041567 13163 catalog_manager.cc:4841] TS 80f2ea5b58fe47a88ca6daca372bcb73 (127.11.40.194:34431): tablet 06d24f907a114b45b287d89694fb4383 (table test-table [id=e89800b73c754a75b71d3eeab7664026]) successfully deleted
I20260929 00:49:55.041839 13818 ts_tablet_manager.cc:1918] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:49:55.041944 13818 log.cc:1196] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/06d24f907a114b45b287d89694fb4383
I20260929 00:49:55.042276 13818 ts_tablet_manager.cc:1939] T 06d24f907a114b45b287d89694fb4383 P 922044b7663748b3a59ba387ae434486: Deleting consensus metadata
I20260929 00:49:55.043962 13161 catalog_manager.cc:4841] TS 922044b7663748b3a59ba387ae434486 (127.11.40.195:45355): tablet 06d24f907a114b45b287d89694fb4383 (table test-table [id=e89800b73c754a75b71d3eeab7664026]) successfully deleted
I20260929 00:49:55.043992 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13389
W20260929 00:49:55.044358 13161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:46801 (error 108)
I20260929 00:49:55.049685 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:55.049218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:46801 (local address 127.0.0.1:48614)
0929 00:49:55.049490 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:55.049587 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:46801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
W20260929 00:49:55.049897 13161 catalog_manager.cc:4584] TS 3c296035b5704f9f92c65a5fcb0e4d36 (127.11.40.193:46801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06d24f907a114b45b287d89694fb4383: Network error: Client connection negotiation failed: client connection to 127.11.40.193:46801: connect: Connection refused (error 111)
I20260929 00:49:55.050761 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13529
I20260929 00:49:55.058637 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13145
2026-09-29T00:49:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6405 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:55 dist-test-slave-5309 krb5kdc[13825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:55 dist-test-slave-5309 krb5kdc[13825](info): set up 2 sockets
Sep 29 00:49:55 dist-test-slave-5309 krb5kdc[13825](info): commencing operation
krb5kdc: starting...
W20260929 00:49:56.110423 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:56.242436 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34737
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:56.353657 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:56.353938 13838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:56.353981 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:56.357911 13838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:56.357970 13838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:56.357990 13838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:56.358006 13838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:56.358027 13838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:56.363466 13838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:56.364660 13838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:56.365788 13838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:56.374958 13846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:56.375066 13843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:56.375018 13838 server_base.cc:1034] running on GCE node
W20260929 00:49:56.375022 13844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:56.375890 13838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:56.377158 13838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:56.378346 13838 hybrid_clock.cc:648] HybridClock initialized: now 1790642996378314 us; error 44 us; skew 500 ppm
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:56.387863 13838 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:56.389061 13838 webserver.cc:460] Webserver started at http://127.11.40.254:41187/ using document root <none> and password file <none>
I20260929 00:49:56.389675 13838 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:56.389770 13838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:56.390017 13838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:56.392241 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "f900f429329140f2aa5f6b65f6b78e74"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "1be8fb11e91225464a5809eebe041361"
server_key_iv: "20be7ce6984c8aa8217c4578f2fe7f94"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.393047 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "f900f429329140f2aa5f6b65f6b78e74"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "1be8fb11e91225464a5809eebe041361"
server_key_iv: "20be7ce6984c8aa8217c4578f2fe7f94"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.397343 13838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:49:56.400171 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.401655 13838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:56.401816 13838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "f900f429329140f2aa5f6b65f6b78e74"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "1be8fb11e91225464a5809eebe041361"
server_key_iv: "20be7ce6984c8aa8217c4578f2fe7f94"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.401943 13838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:56.420951 13838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:56.424166 13838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:56.424502 13838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:56.432337 13838 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34737
I20260929 00:49:56.432368 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34737 every 8 connection(s)
I20260929 00:49:56.433420 13838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:56.436435 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:56.440276 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13838
I20260929 00:49:56.440418 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:56.440704 11427 external_mini_cluster.cc:1428] Setting key 31c2d13bc3380f6c607223c4942e394b
I20260929 00:49:56.442999 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74: Bootstrap starting.
I20260929 00:49:56.445518 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:56.446349 13906 log.cc:824] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:56.448428 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74: No bootstrap required, opened a new log
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:56.452332 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } }
I20260929 00:49:56.452678 13906 raft_consensus.cc:374] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:56.452764 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f900f429329140f2aa5f6b65f6b78e74, State: Initialized, Role: FOLLOWER
I20260929 00:49:56.453339 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } }
I20260929 00:49:56.453474 13906 raft_consensus.cc:388] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:56.453557 13906 raft_consensus.cc:482] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:56.453675 13906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:56.454648 13906 raft_consensus.cc:504] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } }
I20260929 00:49:56.455011 13906 leader_election.cc:302] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f900f429329140f2aa5f6b65f6b78e74; no voters:
I20260929 00:49:56.455338 13906 leader_election.cc:288] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:56.455448 13911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:56.455734 13911 raft_consensus.cc:686] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [term 1 LEADER]: Becoming Leader. State: Replica: f900f429329140f2aa5f6b65f6b78e74, State: Running, Role: LEADER
I20260929 00:49:56.456149 13911 consensus_queue.cc:237] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } }
I20260929 00:49:56.456864 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:56.456960 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:56.442257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.11.40.254:34737)
0929 00:49:56.442768 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:56.442783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:56.442824 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:49:56.443597 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:56.443618 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:56.443987 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:56.444372 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:56.445317 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.446406 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.447304 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.447811 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.447984 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:56.450962 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:56.450968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:56.450984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:56.451023 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:56.453583 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:56.454113 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:56.454119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:56.454125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:56.454251 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:56.454569 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:56.454572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:56.454574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:56.455057 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:56.455262 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:56.455581 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 00:49:56.455915 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":158,"threads_started":1}
I20260929 00:49:56.457399 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f900f429329140f2aa5f6b65f6b78e74. Latest consensus state: current_term: 1 leader_uuid: "f900f429329140f2aa5f6b65f6b78e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } } }
I20260929 00:49:56.457667 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:56.457862 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f900f429329140f2aa5f6b65f6b78e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f900f429329140f2aa5f6b65f6b78e74" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34737 } } }
I20260929 00:49:56.457955 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:56.458211 13918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:56.461396 13918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:49:56.468050 13918 catalog_manager.cc:1347] Generated new cluster ID: ccbc1e25e1af4d08b08fbda86e5f5c07
I20260929 00:49:56.468145 13918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:56.487421 13918 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:56.488830 13918 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:56.515607 13918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f900f429329140f2aa5f6b65f6b78e74: Generated new TSK 0
I20260929 00:49:56.516517 13918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:49:56.557363 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:56.677021 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:56.677328 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:56.677376 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:56.681476 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:56.681543 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:56.681641 13932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:49:56.687196 13932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:56.688344 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:56.689462 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:56.699751 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:56.699790 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:56.699877 13932 server_base.cc:1034] running on GCE node
W20260929 00:49:56.700098 13937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:56.700585 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:56.701321 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:56.702497 13932 hybrid_clock.cc:648] HybridClock initialized: now 1790642996702478 us; error 30 us; skew 500 ppm
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:56.705567 13932 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:49:56.706606 13932 webserver.cc:460] Webserver started at http://127.11.40.193:44865/ using document root <none> and password file <none>
I20260929 00:49:56.707206 13932 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:56.707271 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:56.707495 13932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:56.709761 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "870cc7552aee49578d72da1c2975be39"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "cc4790975ae6292c105acdc0bc7f49fb"
server_key_iv: "2c08e9cf93301ab2fc0778f4f81ccbd9"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.710328 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "870cc7552aee49578d72da1c2975be39"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "cc4790975ae6292c105acdc0bc7f49fb"
server_key_iv: "2c08e9cf93301ab2fc0778f4f81ccbd9"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.714394 13932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:49:56.716961 13947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.718235 13932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:49:56.718361 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "870cc7552aee49578d72da1c2975be39"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "cc4790975ae6292c105acdc0bc7f49fb"
server_key_iv: "2c08e9cf93301ab2fc0778f4f81ccbd9"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.718469 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:56.753001 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:56.756387 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:56.756600 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:56.757346 13932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:56.758350 13932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:56.758402 13932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.758469 13932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:56.758502 13932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.767377 13932 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43655
I20260929 00:49:56.767395 14060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43655 every 8 connection(s)
I20260929 00:49:56.768304 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:49:56.775054 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 13932
I20260929 00:49:56.775156 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:49:56.775486 11427 external_mini_cluster.cc:1428] Setting key e66dbabd70cc03063a70e7ea965563d1
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:56.783685 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:56.770265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:49601 (local address 127.11.40.254:34737)
0929 00:49:56.770474 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:56.770479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:56.771567 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 00:49:56.772462 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:56.772465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:56.772520 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:56.772627 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:56.773868 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.774424 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.775709 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.775953 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.776065 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:56.779307 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:56.779310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:56.779312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:56.779337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:56.781070 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:56.781708 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:56.781717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:56.781719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:56.781816 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:56.782219 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:56.782225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:56.782230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:56.782465 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:56.782550 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:56.783434 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 00:49:56.783557 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 00:49:56.785252 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:56.770715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34737 (local address 127.11.40.193:49601)
0929 00:49:56.771402 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:56.771454 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:56.772255 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:56.772831 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:56.772838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:56.773207 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:56.773715 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:56.773729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.774563 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.774566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:56.775585 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:56.775601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.776113 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.776117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:56.776333 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:56.777144 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:56.777170 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:56.779110 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:56.781237 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:56.781243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:56.781257 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:56.781562 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:56.781928 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:56.781938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:56.781939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:56.782085 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:56.782587 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:56.782597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:56.783032 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 00:49:56.783308 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 00:49:56.783741 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:49:56.786791 14061 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34737
I20260929 00:49:56.787077 14061 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:56.787611 14061 heartbeater.cc:511] Master 127.11.40.254:34737 requested a full tablet report, sending...
I20260929 00:49:56.789390 13865 ts_manager.cc:194] Registered new tserver with Master: 870cc7552aee49578d72da1c2975be39 (127.11.40.193:43655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:49:56.790760 13865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:49601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:49:56.809952 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:56.922833 14066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:56.923127 14066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:56.923171 14066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:56.927520 14066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:56.927590 14066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:56.927685 14066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:49:56.933243 14066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:56.934415 14066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:56.935542 14066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:56.945286 14072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:56.945343 14071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:56.945286 14074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:56.945654 14066 server_base.cc:1034] running on GCE node
I20260929 00:49:56.946086 14066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:56.946748 14066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:56.947927 14066 hybrid_clock.cc:648] HybridClock initialized: now 1790642996947915 us; error 26 us; skew 500 ppm
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:56.950786 14066 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:49:56.951740 14066 webserver.cc:460] Webserver started at http://127.11.40.194:43195/ using document root <none> and password file <none>
I20260929 00:49:56.952330 14066 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:56.952419 14066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:56.952656 14066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:56.954813 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "086cca2b5b2d45a8864df4a676383c3d"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "27f38ad37bbe3a23f8a54e32f9fa25cf"
server_key_iv: "baaca106733570ce3b0761cf955d4a93"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.955380 14066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "086cca2b5b2d45a8864df4a676383c3d"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "27f38ad37bbe3a23f8a54e32f9fa25cf"
server_key_iv: "baaca106733570ce3b0761cf955d4a93"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.959702 14066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:56.962191 14081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.963442 14066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:49:56.963572 14066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "086cca2b5b2d45a8864df4a676383c3d"
format_stamp: "Formatted at 2026-09-29 00:49:56 on dist-test-slave-5309"
server_key: "27f38ad37bbe3a23f8a54e32f9fa25cf"
server_key_iv: "baaca106733570ce3b0761cf955d4a93"
server_key_version: "encryptionkey@0"
I20260929 00:49:56.963682 14066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:56.974259 14066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:56.977624 14066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:56.977826 14066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:56.978418 14066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:56.979256 14066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:56.979322 14066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.979389 14066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:56.979421 14066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:56.988346 14066 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:38355
I20260929 00:49:56.988368 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:38355 every 8 connection(s)
I20260929 00:49:56.989383 14066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:49:56.997000 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14066
I20260929 00:49:56.997156 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:49:56.997463 11427 external_mini_cluster.cc:1428] Setting key 0dd9a0f951941009d28f6418d3d00fe5
Sep 29 00:49:56 dist-test-slave-5309 krb5kdc[13825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642996, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:57.004180 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:56.991348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:53293 (local address 127.11.40.254:34737)
0929 00:49:56.991520 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:56.991525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:56.992606 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 00:49:56.993542 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:56.993545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:56.993591 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:56.993667 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:56.994901 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.995437 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.996569 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.996783 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.996923 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:56.999801 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:56.999805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:56.999813 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:56.999838 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:57.001677 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:57.002370 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:57.002373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:57.002374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:57.002453 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:57.002889 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:57.002892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:57.002893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:57.003079 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:57.003156 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:57.003915 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 00:49:57.004052 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 00:49:57.005764 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:56.991759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34737 (local address 127.11.40.194:53293)
0929 00:49:56.992454 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:56.992498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:56.993343 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:56.993836 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:56.993843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:56.994196 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:56.994758 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:56.994772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.995561 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.995564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:56.996467 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:56.996475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:56.996984 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:56.997000 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:56.997287 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:56.997994 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:56.998016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:56.999629 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:57.001819 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:57.001826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:57.001842 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:57.002261 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:57.002560 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:57.002567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:57.002573 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:57.002795 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:57.003187 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:57.003198 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:57.003597 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 00:49:57.003859 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 00:49:57.004322 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":111,"threads_started":1}
I20260929 00:49:57.007177 14195 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34737
I20260929 00:49:57.007436 14195 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:57.007952 14195 heartbeater.cc:511] Master 127.11.40.254:34737 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:49:57.009177 13865 ts_manager.cc:194] Registered new tserver with Master: 086cca2b5b2d45a8864df4a676383c3d (127.11.40.194:38355)
I20260929 00:49:57.009768 13865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:53293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:49:57.029462 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:57.143227 14200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:57.143559 14200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:57.143647 14200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:57.147898 14200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:57.148008 14200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:57.148102 14200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:49:57.153817 14200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:57.155036 14200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:57.156123 14200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:57.166568 14206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:57.166724 14200 server_base.cc:1034] running on GCE node
W20260929 00:49:57.166568 14208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:57.166566 14205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:57.167332 14200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:57.168150 14200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:57.169337 14200 hybrid_clock.cc:648] HybridClock initialized: now 1790642997169313 us; error 38 us; skew 500 ppm
Sep 29 00:49:57 dist-test-slave-5309 krb5kdc[13825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:57.172698 14200 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:49:57.174137 14200 webserver.cc:460] Webserver started at http://127.11.40.195:36029/ using document root <none> and password file <none>
I20260929 00:49:57.174970 14200 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:57.175071 14200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:57.175395 14200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:57.177918 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "8e5875e2f0f745529c13f84b7b59bcd8"
format_stamp: "Formatted at 2026-09-29 00:49:57 on dist-test-slave-5309"
server_key: "d41748792495e55662933ca4aa2a2cb8"
server_key_iv: "dba844392c374d6c5bff998fdfea42a8"
server_key_version: "encryptionkey@0"
I20260929 00:49:57.178491 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "8e5875e2f0f745529c13f84b7b59bcd8"
format_stamp: "Formatted at 2026-09-29 00:49:57 on dist-test-slave-5309"
server_key: "d41748792495e55662933ca4aa2a2cb8"
server_key_iv: "dba844392c374d6c5bff998fdfea42a8"
server_key_version: "encryptionkey@0"
I20260929 00:49:57.182581 14200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:49:57.185250 14215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:57.186650 14200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:49:57.186762 14200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "8e5875e2f0f745529c13f84b7b59bcd8"
format_stamp: "Formatted at 2026-09-29 00:49:57 on dist-test-slave-5309"
server_key: "d41748792495e55662933ca4aa2a2cb8"
server_key_iv: "dba844392c374d6c5bff998fdfea42a8"
server_key_version: "encryptionkey@0"
I20260929 00:49:57.186933 14200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:57.203544 14200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:57.207607 14200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:57.207836 14200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:57.208518 14200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:49:57.209558 14200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:49:57.209640 14200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:57.209717 14200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:49:57.209758 14200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:57.219218 14200 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:35651
I20260929 00:49:57.219240 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:35651 every 8 connection(s)
I20260929 00:49:57.220188 14200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:49:57.226775 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14200
I20260929 00:49:57.226891 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:49:57.227200 11427 external_mini_cluster.cc:1428] Setting key fe3d62530ebfcf7c48b9168e80000692
Sep 29 00:49:57 dist-test-slave-5309 krb5kdc[13825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:57.235714 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:57.222398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:54287 (local address 127.11.40.254:34737)
0929 00:49:57.222583 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:57.222587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:57.224044 (+ 1457us) server_negotiation.cc:407] Connection header received
0929 00:49:57.225116 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:57.225120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:57.225185 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:57.225287 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:57.226662 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:57.227338 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:57.228744 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:57.229119 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:57.229247 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:57.232179 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:57.232182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:57.232184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:57.232214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:57.233712 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:57.234230 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:57.234233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:57.234234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:57.234311 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:57.234635 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:57.234637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:57.234638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:57.234822 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:57.234899 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:57.235437 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 00:49:57.235584 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 00:49:57.236634 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:57.222813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34737 (local address 127.11.40.195:54287)
0929 00:49:57.223862 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 00:49:57.223909 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:49:57.224778 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:49:57.225480 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:49:57.225488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:49:57.225887 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:49:57.226501 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:57.226516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:57.227470 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:57.227474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:57.228611 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:49:57.228621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:49:57.229293 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:49:57.229296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:49:57.229504 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:57.230132 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:49:57.230149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:49:57.231958 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:49:57.233831 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:57.233837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:57.233852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:57.234136 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:57.234399 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:49:57.234401 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:49:57.234402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:49:57.234551 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:49:57.234906 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:49:57.234912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:49:57.235184 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 00:49:57.235357 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 00:49:57.235681 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":154,"threads_started":1}
I20260929 00:49:57.237924 14329 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34737
I20260929 00:49:57.238219 14329 heartbeater.cc:464] Registering TS with master...
I20260929 00:49:57.238798 14329 heartbeater.cc:511] Master 127.11.40.254:34737 requested a full tablet report, sending...
I20260929 00:49:57.239961 13865 ts_manager.cc:194] Registered new tserver with Master: 8e5875e2f0f745529c13f84b7b59bcd8 (127.11.40.195:35651)
I20260929 00:49:57.240751 13865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:54287
I20260929 00:49:57.241770 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:49:57.256152 13909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:49:57.254494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.11.40.254:34737)
0929 00:49:57.254681 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:57.254686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:57.254830 (+ 144us) server_negotiation.cc:407] Connection header received
0929 00:49:57.255039 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:57.255042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:57.255087 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:57.255178 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:57.255981 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44062: BlockingRecv error: recv got EOF from 127.0.0.1:44062 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 00:49:57.257675 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13932
I20260929 00:49:57.263299 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14066
I20260929 00:49:57.268790 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14200
I20260929 00:49:57.274492 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 13838
2026-09-29T00:49:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2213 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:57 dist-test-slave-5309 krb5kdc[14341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:57 dist-test-slave-5309 krb5kdc[14341](info): set up 2 sockets
Sep 29 00:49:57 dist-test-slave-5309 krb5kdc[14341](info): commencing operation
krb5kdc: starting...
W20260929 00:49:58.322326 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:58.452442 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43645
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:49:58.564229 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:49:58.564566 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:49:58.564640 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:49:58.568706 14354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:49:58.568806 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:49:58.568866 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:49:58.568903 14354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:49:58.568934 14354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:49:58.574271 14354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:49:58.575489 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:49:58.576576 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:49:58.585639 14359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:58.585634 14360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:49:58.585683 14362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:49:58.586014 14354 server_base.cc:1034] running on GCE node
I20260929 00:49:58.586614 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:49:58.587811 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:49:58.588981 14354 hybrid_clock.cc:648] HybridClock initialized: now 1790642998588965 us; error 29 us; skew 500 ppm
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642998, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:49:58.591843 14354 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:49:58.592756 14354 webserver.cc:460] Webserver started at http://127.11.40.254:35155/ using document root <none> and password file <none>
I20260929 00:49:58.593339 14354 fs_manager.cc:359] Metadata directory not provided
I20260929 00:49:58.593406 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:49:58.593575 14354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:49:58.595654 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "34b98f921c554542b5ad616cc63b45e6"
format_stamp: "Formatted at 2026-09-29 00:49:58 on dist-test-slave-5309"
server_key: "d07be4c039848f546012623e7da1136e"
server_key_iv: "3765ac2070ecfb1e2f0512299dde14e6"
server_key_version: "encryptionkey@0"
I20260929 00:49:58.596208 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "34b98f921c554542b5ad616cc63b45e6"
format_stamp: "Formatted at 2026-09-29 00:49:58 on dist-test-slave-5309"
server_key: "d07be4c039848f546012623e7da1136e"
server_key_iv: "3765ac2070ecfb1e2f0512299dde14e6"
server_key_version: "encryptionkey@0"
I20260929 00:49:58.600206 14354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:49:58.603158 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:49:58.604655 14354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:49:58.604813 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "34b98f921c554542b5ad616cc63b45e6"
format_stamp: "Formatted at 2026-09-29 00:49:58 on dist-test-slave-5309"
server_key: "d07be4c039848f546012623e7da1136e"
server_key_iv: "3765ac2070ecfb1e2f0512299dde14e6"
server_key_version: "encryptionkey@0"
I20260929 00:49:58.604985 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:49:58.628029 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:49:58.631803 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:49:58.631970 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:49:58.639398 14354 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:43645
I20260929 00:49:58.639695 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:43645 every 8 connection(s)
I20260929 00:49:58.640455 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:49:58.643611 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:49:58.649535 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14354
I20260929 00:49:58.649691 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:49:58.649752 14422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: Bootstrap starting.
I20260929 00:49:58.649996 11427 external_mini_cluster.cc:1428] Setting key fa51ceea13aea57e4a384814578b3944
I20260929 00:49:58.652696 14422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: Neither blocks nor log segments found. Creating new log.
I20260929 00:49:58.653692 14422 log.cc:824] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: Log is configured to *not* fsync() on all Append() calls
I20260929 00:49:58.656224 14422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: No bootstrap required, opened a new log
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:49:58.660935 14422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } }
I20260929 00:49:58.661211 14422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:49:58.661304 14422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b98f921c554542b5ad616cc63b45e6, State: Initialized, Role: FOLLOWER
I20260929 00:49:58.661808 14422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } }
I20260929 00:49:58.661944 14422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:49:58.662014 14422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:49:58.662117 14422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:49:58.663084 14422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } }
I20260929 00:49:58.663506 14422 leader_election.cc:302] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b98f921c554542b5ad616cc63b45e6; no voters:
I20260929 00:49:58.663929 14422 leader_election.cc:288] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:49:58.664036 14427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:49:58.664250 14427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [term 1 LEADER]: Becoming Leader. State: Replica: 34b98f921c554542b5ad616cc63b45e6, State: Running, Role: LEADER
I20260929 00:49:58.664564 14427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } }
I20260929 00:49:58.665403 14422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:49:58.666266 14428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b98f921c554542b5ad616cc63b45e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } } }
I20260929 00:49:58.666405 14428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:58.666692 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:58.651417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.11.40.254:43645)
0929 00:49:58.651904 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:58.651927 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:58.651976 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:49:58.652983 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:58.653013 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:58.653464 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:58.653877 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:58.654841 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:58.656284 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:58.657294 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:58.658033 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:58.658214 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:58.661039 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:58.661045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:58.661058 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:49:58.661092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:58.663032 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:58.663560 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:58.663565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:58.663570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:58.663725 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:49:58.664121 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:49:58.664127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:49:58.664131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:49:58.664659 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:58.664917 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:58.665297 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 00:49:58.665619 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20260929 00:49:58.666728 14429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b98f921c554542b5ad616cc63b45e6. Latest consensus state: current_term: 1 leader_uuid: "34b98f921c554542b5ad616cc63b45e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b98f921c554542b5ad616cc63b45e6" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43645 } } }
I20260929 00:49:58.666826 14429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6 [sys.catalog]: This master's current role is: LEADER
I20260929 00:49:58.669912 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 00:49:58.670696 14442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:49:58.670780 14442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:49:58.670871 14436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:49:58.671548 14436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:49:58.677927 14436 catalog_manager.cc:1347] Generated new cluster ID: 85448f55fcc04cb4a975baa6eb093e77
I20260929 00:49:58.677996 14436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:49:58.689937 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:49:58.684144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.11.40.254:43645)
0929 00:49:58.684265 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 00:49:58.684270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:49:58.684548 (+ 278us) server_negotiation.cc:407] Connection header received
0929 00:49:58.684683 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:49:58.684685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:49:58.684742 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:49:58.684851 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:49:58.685460 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:58.686233 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:58.687429 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:49:58.687788 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:49:58.687877 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:49:58.688848 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:49:58.688853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:49:58.688856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:49:58.689452 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:49:58.689497 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:49:58.689517 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:49:58.689598 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:49:58.689724 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 00:49:58.689786 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:49:58.701584 14436 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:49:58.702913 14436 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:49:58.709838 14436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b98f921c554542b5ad616cc63b45e6: Generated new TSK 0
I20260929 00:49:58.710602 14436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:49:58.719177 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14354
2026-09-29T00:49:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1442 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14455](info): set up 2 sockets
Sep 29 00:49:58 dist-test-slave-5309 krb5kdc[14455](info): commencing operation
krb5kdc: starting...
W20260929 00:49:59.766359 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:49:59 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:49:59.901202 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39639
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:00.017745 14468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:00.018057 14468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:00.018138 14468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:00.022341 14468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:00.022425 14468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:00.022445 14468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:00.022464 14468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:00.022480 14468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:00.028539 14468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:00.029883 14468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:00.031013 14468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:00.040338 14473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.040477 14468 server_base.cc:1034] running on GCE node
W20260929 00:50:00.040374 14474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.040378 14476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.041275 14468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:00.042483 14468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:00.043676 14468 hybrid_clock.cc:648] HybridClock initialized: now 1790643000043660 us; error 28 us; skew 500 ppm
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:00.046598 14468 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:00.047514 14468 webserver.cc:460] Webserver started at http://127.11.40.254:34635/ using document root <none> and password file <none>
I20260929 00:50:00.048084 14468 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:00.048170 14468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:00.048476 14468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:00.050691 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "9a5d43542890406792ed3f44c7b3c9a8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "4bb9ad4b015766be93befc3517294733"
server_key_iv: "5788b917ecb3fabfc1c24f6998751d16"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.051283 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "9a5d43542890406792ed3f44c7b3c9a8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "4bb9ad4b015766be93befc3517294733"
server_key_iv: "5788b917ecb3fabfc1c24f6998751d16"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.055297 14468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:00.058034 14483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.059494 14468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:00.059660 14468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "9a5d43542890406792ed3f44c7b3c9a8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "4bb9ad4b015766be93befc3517294733"
server_key_iv: "5788b917ecb3fabfc1c24f6998751d16"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.059784 14468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:00.070205 14468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:00.073184 14468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:00.073503 14468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:00.081434 14468 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:39639
I20260929 00:50:00.081784 14535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:39639 every 8 connection(s)
I20260929 00:50:00.082480 14468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:00.085412 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:00.088295 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14468
I20260929 00:50:00.088433 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:00.088735 11427 external_mini_cluster.cc:1428] Setting key 619387612b7d4c94b994d61f3d036d19
I20260929 00:50:00.091856 14536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8: Bootstrap starting.
I20260929 00:50:00.094591 14536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:00.095494 14536 log.cc:824] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:00.097981 14536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8: No bootstrap required, opened a new log
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790642999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:00.102368 14536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } }
I20260929 00:50:00.102775 14536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:00.102909 14536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5d43542890406792ed3f44c7b3c9a8, State: Initialized, Role: FOLLOWER
I20260929 00:50:00.103525 14536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } }
I20260929 00:50:00.103693 14536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:00.103782 14536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:00.103896 14536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:00.103996 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.090272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.11.40.254:39639)
0929 00:50:00.090720 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:00.090736 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:00.090774 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:50:00.091427 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:00.091446 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:00.091834 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:00.092195 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:00.093126 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.094147 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.095347 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.095905 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.096102 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.098811 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:00.098816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:00.098830 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:00.098856 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:00.100565 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.101105 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.101110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.101114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.101225 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.101504 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.101505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.101506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.101875 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:00.102174 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:00.102530 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 00:50:00.102856 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260929 00:50:00.104938 14536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } }
I20260929 00:50:00.105285 14536 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5d43542890406792ed3f44c7b3c9a8; no voters:
I20260929 00:50:00.105626 14536 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:00.105698 14541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:00.105909 14541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5d43542890406792ed3f44c7b3c9a8, State: Running, Role: LEADER
I20260929 00:50:00.106277 14541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } }
I20260929 00:50:00.106954 14536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:00.107999 14542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5d43542890406792ed3f44c7b3c9a8. Latest consensus state: current_term: 1 leader_uuid: "9a5d43542890406792ed3f44c7b3c9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } } }
I20260929 00:50:00.108141 14542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:00.108456 14543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5d43542890406792ed3f44c7b3c9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5d43542890406792ed3f44c7b3c9a8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39639 } } }
I20260929 00:50:00.108546 14543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:00.109531 14550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:00.112880 14550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:00.118719 14550 catalog_manager.cc:1347] Generated new cluster ID: e80f2bbbb71548dd99464c5b1584ea36
I20260929 00:50:00.118865 14550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:00.128953 14550 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:00.130031 14550 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:00.138407 14550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5d43542890406792ed3f44c7b3c9a8: Generated new TSK 0
I20260929 00:50:00.139214 14550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:00.181151 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:00.302292 14562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:00.302632 14562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:00.302709 14562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:00.306979 14562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:00.307075 14562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:00.307165 14562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:00.313004 14562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:00.314177 14562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:00.315254 14562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:00.325701 14567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.325712 14568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.325708 14570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.326129 14562 server_base.cc:1034] running on GCE node
I20260929 00:50:00.326556 14562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:00.327246 14562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:00.328460 14562 hybrid_clock.cc:648] HybridClock initialized: now 1790643000328438 us; error 39 us; skew 500 ppm
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:00.331554 14562 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:00.332520 14562 webserver.cc:460] Webserver started at http://127.11.40.193:33035/ using document root <none> and password file <none>
I20260929 00:50:00.333243 14562 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:00.333356 14562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:00.333578 14562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:00.335638 14562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "39f1f91f31f741008a8a844493ca75d5"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "3005da005ee5b191b61f044f1c0eec6a"
server_key_iv: "43a8e4e26f4c72e7f1b453409f39d91b"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.336199 14562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "39f1f91f31f741008a8a844493ca75d5"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "3005da005ee5b191b61f044f1c0eec6a"
server_key_iv: "43a8e4e26f4c72e7f1b453409f39d91b"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.339968 14562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:00.342555 14577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.343833 14562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:00.343981 14562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "39f1f91f31f741008a8a844493ca75d5"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "3005da005ee5b191b61f044f1c0eec6a"
server_key_iv: "43a8e4e26f4c72e7f1b453409f39d91b"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.344099 14562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:00.361819 14562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:00.365424 14562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:00.365696 14562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:00.366343 14562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:00.367213 14562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:00.367285 14562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.367357 14562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:00.367404 14562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.376843 14562 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:38327
I20260929 00:50:00.376884 14690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:38327 every 8 connection(s)
I20260929 00:50:00.377803 14562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:00.379457 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14562
I20260929 00:50:00.379606 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:00.379989 11427 external_mini_cluster.cc:1428] Setting key 1a2ff02a74cf9bbb9c352e653624c640
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:00.393869 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.379769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:38145 (local address 127.11.40.254:39639)
0929 00:50:00.379941 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:00.379946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:00.380917 (+ 971us) server_negotiation.cc:407] Connection header received
0929 00:50:00.382527 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:00.382534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:00.382597 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:00.382692 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:00.384299 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.385181 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.386451 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.386916 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.387058 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.389976 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:00.389980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:00.389985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:00.390018 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:00.391552 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.392082 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.392085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.392086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.392163 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.392583 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.392586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.392587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.392774 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:00.392944 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:00.393616 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 00:50:00.393741 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:50:00.395287 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.380232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39639 (local address 127.11.40.193:38145)
0929 00:50:00.380747 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:00.380810 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:00.382303 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:00.382851 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:00.382858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:00.383394 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:00.384117 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.384141 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.385300 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.385303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.386328 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.386337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.387098 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.387101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.387256 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.388030 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:00.388049 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:00.389772 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:00.391671 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.391677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.391692 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.391991 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.392262 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.392270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.392273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.392495 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.392977 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:00.392984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:00.393331 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 00:50:00.393574 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 00:50:00.394025 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260929 00:50:00.397222 14691 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39639
I20260929 00:50:00.397562 14691 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:00.398190 14691 heartbeater.cc:511] Master 127.11.40.254:39639 requested a full tablet report, sending...
I20260929 00:50:00.399964 14500 ts_manager.cc:194] Registered new tserver with Master: 39f1f91f31f741008a8a844493ca75d5 (127.11.40.193:38327)
I20260929 00:50:00.401262 14500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:38145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:00.412776 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:00.538810 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:00.539248 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:00.539367 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:00.543551 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:00.543699 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:00.543819 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:00.549829 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:00.551043 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:00.552148 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:00.562989 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.562971 14702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.563151 14696 server_base.cc:1034] running on GCE node
W20260929 00:50:00.563318 14704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.563827 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:00.564635 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:00.565834 14696 hybrid_clock.cc:648] HybridClock initialized: now 1790643000565811 us; error 31 us; skew 500 ppm
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:00.568939 14696 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:00.570009 14696 webserver.cc:460] Webserver started at http://127.11.40.194:43021/ using document root <none> and password file <none>
I20260929 00:50:00.570626 14696 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:00.570693 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:00.570924 14696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:00.573493 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "5d7aaee0df0a4196b1c7a952c61e1e9a"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "64de86642325cbce29fabd05f7c4441b"
server_key_iv: "0368d992c9154d4d88562398eac35e05"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.574112 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "5d7aaee0df0a4196b1c7a952c61e1e9a"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "64de86642325cbce29fabd05f7c4441b"
server_key_iv: "0368d992c9154d4d88562398eac35e05"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.578261 14696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:00.581849 14711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.583314 14696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:00.583479 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "5d7aaee0df0a4196b1c7a952c61e1e9a"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "64de86642325cbce29fabd05f7c4441b"
server_key_iv: "0368d992c9154d4d88562398eac35e05"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.583609 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:00.618115 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:00.621699 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:00.621922 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:00.622558 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:00.623440 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:00.623512 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.623582 14696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:00.623636 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.633854 14696 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:46719
I20260929 00:50:00.633900 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:46719 every 8 connection(s)
I20260929 00:50:00.634807 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:00.640998 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14696
I20260929 00:50:00.641161 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:00.641438 11427 external_mini_cluster.cc:1428] Setting key 4ef4ac4e090fe1e403d0972fddee6e31
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:00.651083 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.636855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:41569 (local address 127.11.40.254:39639)
0929 00:50:00.637062 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:00.637068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:00.637876 (+ 808us) server_negotiation.cc:407] Connection header received
0929 00:50:00.639479 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:00.639486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:00.639554 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:00.639656 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:00.641074 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.641750 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.643141 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.643382 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.643503 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.646426 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:00.646432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:00.646438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:00.646473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:00.648451 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.649125 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.649131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.649133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.649237 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.649617 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.649621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.649622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.649900 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:00.650018 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:00.650702 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 00:50:00.650857 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:00.651980 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.637240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39639 (local address 127.11.40.194:41569)
0929 00:50:00.637709 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:00.637765 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:00.639184 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:00.639867 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:00.639876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:00.640354 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:00.640929 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.640946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.641880 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.641884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.643013 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.643023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.643555 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.643559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.643727 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.644408 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:00.644437 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:00.646230 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:00.648606 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.648611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.648626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.648984 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.649339 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.649341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.649343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.649483 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.650035 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:00.650040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:00.650311 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 00:50:00.650552 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 00:50:00.650889 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260929 00:50:00.653509 14825 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39639
I20260929 00:50:00.653838 14825 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:00.654469 14825 heartbeater.cc:511] Master 127.11.40.254:39639 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:50:00.655762 14500 ts_manager.cc:194] Registered new tserver with Master: 5d7aaee0df0a4196b1c7a952c61e1e9a (127.11.40.194:46719)
I20260929 00:50:00.656635 14500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:41569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:00.675258 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:00.787806 14830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:00.788161 14830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:00.788259 14830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:00.792423 14830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:00.792526 14830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:00.792626 14830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:00.798050 14830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:00.799187 14830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:00.800271 14830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:00.810524 14835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.810571 14836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:00.810632 14838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:00.810791 14830 server_base.cc:1034] running on GCE node
I20260929 00:50:00.811247 14830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:00.811980 14830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:00.813184 14830 hybrid_clock.cc:648] HybridClock initialized: now 1790643000813128 us; error 71 us; skew 500 ppm
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:00.816380 14830 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:00.817436 14830 webserver.cc:460] Webserver started at http://127.11.40.195:32961/ using document root <none> and password file <none>
I20260929 00:50:00.818038 14830 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:00.818135 14830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:00.818372 14830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:00.820533 14830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "6dfa339e8e7e45c490527a5de29b5bf8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "31c8829e4b6f2a90d5c78d866d4286b7"
server_key_iv: "dc5b688775117e1e8bf0989bd7fb5134"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.821141 14830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "6dfa339e8e7e45c490527a5de29b5bf8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "31c8829e4b6f2a90d5c78d866d4286b7"
server_key_iv: "dc5b688775117e1e8bf0989bd7fb5134"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.825119 14830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:00.827777 14845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.829381 14830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:00.829538 14830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "6dfa339e8e7e45c490527a5de29b5bf8"
format_stamp: "Formatted at 2026-09-29 00:50:00 on dist-test-slave-5309"
server_key: "31c8829e4b6f2a90d5c78d866d4286b7"
server_key_iv: "dc5b688775117e1e8bf0989bd7fb5134"
server_key_version: "encryptionkey@0"
I20260929 00:50:00.829680 14830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:00.847527 14830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:00.851395 14830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:00.851627 14830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:00.852269 14830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:00.853222 14830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:00.853295 14830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.853379 14830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:00.853433 14830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:00.863555 14830 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:33011
I20260929 00:50:00.863575 14958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:33011 every 8 connection(s)
I20260929 00:50:00.864635 14830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:00.872460 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14830
I20260929 00:50:00.872624 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:00.872960 11427 external_mini_cluster.cc:1428] Setting key 1be2a8b4614500baffeda7ac4768ac9d
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:00.880966 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.867073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:44447 (local address 127.11.40.254:39639)
0929 00:50:00.867260 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:00.867264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:00.868296 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 00:50:00.869829 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:00.869833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:00.869895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:00.870005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:00.871262 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.871849 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.873560 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.873804 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.873927 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.877305 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:00.877308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:00.877310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:00.877336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:00.878733 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.879317 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.879321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.879322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.879403 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.879787 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.879791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.879792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.879992 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:00.880103 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:00.880622 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 00:50:00.880758 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 00:50:00.881832 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.867539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39639 (local address 127.11.40.195:44447)
0929 00:50:00.868109 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:00.868167 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:00.869589 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:00.870194 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:00.870202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:00.870589 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:00.871105 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.871119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.872002 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.872006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.873426 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:00.873436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.873973 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.873986 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:00.874158 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.875487 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:00.875510 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:00.877127 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:00.878890 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.878896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.878920 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.879200 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.879526 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:00.879529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:00.879531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:00.879685 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:00.880100 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:00.880107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:00.880377 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 00:50:00.880570 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 00:50:00.880961 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":144,"threads_started":1}
I20260929 00:50:00.883054 14959 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39639
I20260929 00:50:00.883283 14959 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:00.883752 14959 heartbeater.cc:511] Master 127.11.40.254:39639 requested a full tablet report, sending...
I20260929 00:50:00.885008 14500 ts_manager.cc:194] Registered new tserver with Master: 6dfa339e8e7e45c490527a5de29b5bf8 (127.11.40.195:33011)
I20260929 00:50:00.885576 14500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:44447
I20260929 00:50:00.887528 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643000, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:00.914655 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:00.905771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.11.40.254:39639)
0929 00:50:00.906031 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:00.906036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:00.906196 (+ 160us) server_negotiation.cc:407] Connection header received
0929 00:50:00.906370 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:00.906374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:00.906419 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:00.906523 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:00.907276 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.907872 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.908826 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:00.909063 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:00.909176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:00.911568 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:00.911571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:00.911573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:00.911598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:00.912961 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.913373 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.913378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.913380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.913462 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:00.913739 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:00.913743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:00.913744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:00.913972 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:00.914112 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:00.914348 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 00:50:00.914471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
W20260929 00:50:00.915088 14500 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32846
I20260929 00:50:00.916004 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14562
I20260929 00:50:00.921743 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14696
I20260929 00:50:00.927120 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14830
I20260929 00:50:00.933104 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14468
2026-09-29T00:50:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2215 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14971](info): set up 2 sockets
Sep 29 00:50:00 dist-test-slave-5309 krb5kdc[14971](info): commencing operation
krb5kdc: starting...
W20260929 00:50:01.986336 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:50:02.128245 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:32839
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:02.263744 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:02.264120 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:02.264199 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:02.268540 14984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:02.268661 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:02.268695 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:02.268736 14984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:02.268767 14984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:02.274845 14984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:32839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:02.276976 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:02.278262 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:02.287632 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:02.287626 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:02.287626 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:02.287804 14984 server_base.cc:1034] running on GCE node
I20260929 00:50:02.288571 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:02.289851 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:02.291069 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790643002291052 us; error 31 us; skew 500 ppm
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:02.294116 14984 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:02.295212 14984 webserver.cc:460] Webserver started at http://127.11.40.254:40657/ using document root <none> and password file <none>
I20260929 00:50:02.295795 14984 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:02.295890 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:02.296123 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:02.298331 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "5a502d40fbd94eb99f507224418e8a7c"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "103abf8473af891e98d076e854a938bd"
server_key_iv: "15bce28516d7ff0470fac88412aeb42d"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.298971 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "5a502d40fbd94eb99f507224418e8a7c"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "103abf8473af891e98d076e854a938bd"
server_key_iv: "15bce28516d7ff0470fac88412aeb42d"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.303072 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:02.305737 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.307042 14984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:02.307199 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "5a502d40fbd94eb99f507224418e8a7c"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "103abf8473af891e98d076e854a938bd"
server_key_iv: "15bce28516d7ff0470fac88412aeb42d"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.307324 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:02.335551 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:02.339417 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:02.339635 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:02.347492 14984 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:32839
I20260929 00:50:02.347527 15051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:32839 every 8 connection(s)
I20260929 00:50:02.348516 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:02.351482 15052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:02.356447 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 14984
I20260929 00:50:02.356556 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:02.356925 11427 external_mini_cluster.cc:1428] Setting key 3a1095ae5985a334b2fa5cc27e831297
I20260929 00:50:02.357729 15052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c: Bootstrap starting.
I20260929 00:50:02.360493 15052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:02.361735 15052 log.cc:824] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:02.364202 15052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c: No bootstrap required, opened a new log
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:02.369500 15052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } }
I20260929 00:50:02.369764 15052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:02.369844 15052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a502d40fbd94eb99f507224418e8a7c, State: Initialized, Role: FOLLOWER
I20260929 00:50:02.370431 15052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } }
I20260929 00:50:02.370626 15052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:02.370713 15052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:02.370841 15052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:02.372079 15052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } }
I20260929 00:50:02.372517 15052 leader_election.cc:302] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a502d40fbd94eb99f507224418e8a7c; no voters:
I20260929 00:50:02.372953 15052 leader_election.cc:288] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:02.373057 15057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:02.373317 15057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [term 1 LEADER]: Becoming Leader. State: Replica: 5a502d40fbd94eb99f507224418e8a7c, State: Running, Role: LEADER
I20260929 00:50:02.373644 15057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } }
I20260929 00:50:02.373888 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:02.358512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.11.40.254:32839)
0929 00:50:02.359008 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:02.359031 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:02.359078 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:50:02.359920 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:02.359943 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:02.360386 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:02.360963 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:02.361619 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.363165 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.364125 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.364941 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.365131 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:02.368161 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:02.368166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:02.368181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:02.368210 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:02.370223 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.370776 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.370787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.370791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.370983 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.371317 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.371319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.371320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.371724 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:02.371884 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:02.372306 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 00:50:02.372653 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":133,"threads_started":1}
I20260929 00:50:02.374432 15052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:02.374924 15057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a502d40fbd94eb99f507224418e8a7c. Latest consensus state: current_term: 1 leader_uuid: "5a502d40fbd94eb99f507224418e8a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } } }
I20260929 00:50:02.375051 15057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:02.375380 15058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a502d40fbd94eb99f507224418e8a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a502d40fbd94eb99f507224418e8a7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 32839 } } }
I20260929 00:50:02.375484 15058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:02.376513 15061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:02.379863 15061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:02.386132 15061 catalog_manager.cc:1347] Generated new cluster ID: 456e4bd955ba4726912b6f3155b79755
I20260929 00:50:02.386234 15061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:02.416179 15061 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:02.417418 15061 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:02.422950 15061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a502d40fbd94eb99f507224418e8a7c: Generated new TSK 0
I20260929 00:50:02.423751 15061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:02.464480 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:02.596299 15078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:02.596608 15078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:02.596657 15078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:02.601029 15078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:02.601106 15078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:02.601198 15078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:02.606761 15078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:02.607913 15078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:02.609098 15078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:02.619462 15083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:02.619552 15086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:02.619618 15078 server_base.cc:1034] running on GCE node
W20260929 00:50:02.619565 15084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:02.620312 15078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:02.621111 15078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:02.622401 15078 hybrid_clock.cc:648] HybridClock initialized: now 1790643002622302 us; error 44 us; skew 500 ppm
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:02.625599 15078 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:02.626605 15078 webserver.cc:460] Webserver started at http://127.11.40.193:40805/ using document root <none> and password file <none>
I20260929 00:50:02.627233 15078 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:02.627341 15078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:02.627689 15078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:02.630020 15078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "ebbfd31b4f874fe387e0a3696bfafb82"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "525c6fbb58a70c213391fbdc751d91f0"
server_key_iv: "675be3f6785da46c00a0469680021de2"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.630627 15078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "ebbfd31b4f874fe387e0a3696bfafb82"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "525c6fbb58a70c213391fbdc751d91f0"
server_key_iv: "675be3f6785da46c00a0469680021de2"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.634896 15078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:02.637951 15093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.639658 15078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 00:50:02.639818 15078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "ebbfd31b4f874fe387e0a3696bfafb82"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "525c6fbb58a70c213391fbdc751d91f0"
server_key_iv: "675be3f6785da46c00a0469680021de2"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.639963 15078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:02.654351 15078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:02.657784 15078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:02.657944 15078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:02.658490 15078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:02.659297 15078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:02.659345 15078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.659386 15078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:02.659401 15078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.668498 15078 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:39215
I20260929 00:50:02.668577 15206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:39215 every 8 connection(s)
I20260929 00:50:02.669533 15078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:02.672495 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15078
I20260929 00:50:02.672632 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:02.673005 11427 external_mini_cluster.cc:1428] Setting key 78764591728d260b19bbd1f65f37bbda
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:02.685637 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:02.671517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:53933 (local address 127.11.40.254:32839)
0929 00:50:02.671658 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:02.671661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:02.672759 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 00:50:02.674690 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:02.674694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:02.674746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:02.674836 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:02.676274 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.677062 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.678371 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.678636 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.678764 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:02.681574 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:02.681577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:02.681579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:02.681605 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:02.683255 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.683852 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.683858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.683860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.683948 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.684378 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.684382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.684384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.684587 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:02.684679 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:02.685317 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 00:50:02.685468 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:02.686735 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:02.671954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:32839 (local address 127.11.40.193:53933)
0929 00:50:02.672534 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:02.672639 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:02.674449 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:02.675023 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:02.675031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:02.675518 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:02.676114 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:02.676129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.677199 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.677203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:02.678182 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:02.678191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.678814 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.678819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:02.679021 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:02.679711 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:02.679729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:02.681379 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:02.683393 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:02.683399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:02.683417 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:02.683730 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:02.684051 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:02.684055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:02.684058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:02.684232 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:02.684702 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:02.684708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:02.685069 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 00:50:02.685262 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 00:50:02.685640 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":146,"threads_started":1}
I20260929 00:50:02.688446 15207 heartbeater.cc:347] Connected to a master server at 127.11.40.254:32839
I20260929 00:50:02.688794 15207 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:02.689633 15207 heartbeater.cc:511] Master 127.11.40.254:32839 requested a full tablet report, sending...
I20260929 00:50:02.691682 15016 ts_manager.cc:194] Registered new tserver with Master: ebbfd31b4f874fe387e0a3696bfafb82 (127.11.40.193:39215)
I20260929 00:50:02.693542 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:53933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:02.709350 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:02.826924 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:02.827210 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:02.827255 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:02.831367 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:02.831447 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:02.831511 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:02.836906 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:02.837977 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:02.838975 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:02.849356 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:02.849455 15212 server_base.cc:1034] running on GCE node
W20260929 00:50:02.849356 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:02.849370 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:02.850147 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:02.850873 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:02.852069 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790643002852050 us; error 33 us; skew 500 ppm
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:02.854929 15212 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:02.855926 15212 webserver.cc:460] Webserver started at http://127.11.40.194:45285/ using document root <none> and password file <none>
I20260929 00:50:02.856526 15212 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:02.856622 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:02.856932 15212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:02.859068 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "d33b19dc7d464a8bb465869c0f4221f5"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "e9c0404cc8fbcac2bda0ef8dd0812cbb"
server_key_iv: "9942c1e42c32b505f4249a4859527387"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.859642 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "d33b19dc7d464a8bb465869c0f4221f5"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "e9c0404cc8fbcac2bda0ef8dd0812cbb"
server_key_iv: "9942c1e42c32b505f4249a4859527387"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.863557 15212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:02.866158 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.867414 15212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:50:02.867588 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "d33b19dc7d464a8bb465869c0f4221f5"
format_stamp: "Formatted at 2026-09-29 00:50:02 on dist-test-slave-5309"
server_key: "e9c0404cc8fbcac2bda0ef8dd0812cbb"
server_key_iv: "9942c1e42c32b505f4249a4859527387"
server_key_version: "encryptionkey@0"
I20260929 00:50:02.867708 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:02.890060 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:02.893114 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:02.893322 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:02.893927 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:02.894771 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:02.894841 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.894910 15212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:02.894964 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:02.904726 15212 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:42623
I20260929 00:50:02.904915 15340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:42623 every 8 connection(s)
I20260929 00:50:02.905767 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:02.906769 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15212
I20260929 00:50:02.906888 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:02.907177 11427 external_mini_cluster.cc:1428] Setting key c3ea6a66e2d1e0e8978ac5a7faab0691
Sep 29 00:50:02 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:50:02.921564 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:02.907905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:42649 (local address 127.11.40.254:32839)
0929 00:50:02.908043 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:02.908047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:02.909307 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 00:50:02.911391 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:02.911398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:02.911463 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:02.911556 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:02.912794 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.913413 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.914628 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.914853 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.914968 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:02.917748 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:02.917751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:02.917753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:02.917790 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:02.919239 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.919752 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.919757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.919759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.919844 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:02.920231 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:02.920236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:02.920238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:02.920472 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:02.920581 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:02.921242 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 00:50:02.921367 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:50:02.922680 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:02.908453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:32839 (local address 127.11.40.194:42649)
0929 00:50:02.909110 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:02.909174 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:02.911106 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:02.911757 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:02.911765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:02.912161 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:02.912658 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:02.912672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.913552 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.913556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:02.914465 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:02.914505 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:02.915014 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:02.915018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:02.915199 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:02.915863 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:02.915880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:02.917562 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:02.919354 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:02.919360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:02.919377 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:02.919647 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:02.919939 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:02.919944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:02.919946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:02.920113 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:02.920592 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:02.920598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:02.920915 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 00:50:02.921179 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 00:50:02.921562 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":150,"threads_started":1}
I20260929 00:50:02.924120 15341 heartbeater.cc:347] Connected to a master server at 127.11.40.254:32839
I20260929 00:50:02.924378 15341 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:02.924975 15341 heartbeater.cc:511] Master 127.11.40.254:32839 requested a full tablet report, sending...
I20260929 00:50:02.926159 15016 ts_manager.cc:194] Registered new tserver with Master: d33b19dc7d464a8bb465869c0f4221f5 (127.11.40.194:42623)
I20260929 00:50:02.927085 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:42649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:02.941411 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:03.061885 15346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:03.062168 15346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:03.062213 15346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:03.066325 15346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:03.066392 15346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:03.066454 15346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:03.071894 15346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:32839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:03.073006 15346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:03.074044 15346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:03.084503 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:03.084493 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:03.084493 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:03.084820 15346 server_base.cc:1034] running on GCE node
I20260929 00:50:03.085342 15346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:03.086045 15346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:03.087261 15346 hybrid_clock.cc:648] HybridClock initialized: now 1790643003087241 us; error 33 us; skew 500 ppm
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643003, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:03.090662 15346 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:03.091632 15346 webserver.cc:460] Webserver started at http://127.11.40.195:45283/ using document root <none> and password file <none>
I20260929 00:50:03.092245 15346 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:03.092339 15346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:03.092577 15346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:03.094801 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "18c877d121f243348a5d5b4537230d3d"
format_stamp: "Formatted at 2026-09-29 00:50:03 on dist-test-slave-5309"
server_key: "f17dacdb66ab1b5179a6f87eabb4352c"
server_key_iv: "a8d75c177714e85f3ddfbc0705a49b8b"
server_key_version: "encryptionkey@0"
I20260929 00:50:03.095417 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "18c877d121f243348a5d5b4537230d3d"
format_stamp: "Formatted at 2026-09-29 00:50:03 on dist-test-slave-5309"
server_key: "f17dacdb66ab1b5179a6f87eabb4352c"
server_key_iv: "a8d75c177714e85f3ddfbc0705a49b8b"
server_key_version: "encryptionkey@0"
I20260929 00:50:03.099900 15346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:03.102784 15361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:03.104202 15346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:03.104377 15346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "18c877d121f243348a5d5b4537230d3d"
format_stamp: "Formatted at 2026-09-29 00:50:03 on dist-test-slave-5309"
server_key: "f17dacdb66ab1b5179a6f87eabb4352c"
server_key_iv: "a8d75c177714e85f3ddfbc0705a49b8b"
server_key_version: "encryptionkey@0"
I20260929 00:50:03.104555 15346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:03.122391 15346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:03.126232 15346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:03.126487 15346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:03.127143 15346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:03.128041 15346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:03.128110 15346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:03.128186 15346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:03.128227 15346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:03.137900 15346 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:35453
I20260929 00:50:03.137928 15474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:35453 every 8 connection(s)
I20260929 00:50:03.138882 15346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643003, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:03.149184 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15346
I20260929 00:50:03.149331 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:03.149619 11427 external_mini_cluster.cc:1428] Setting key db5786f14c81317b538cd254819e1f06
I20260929 00:50:03.153999 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:03.140979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:55719 (local address 127.11.40.254:32839)
0929 00:50:03.141165 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:03.141169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:03.142028 (+ 859us) server_negotiation.cc:407] Connection header received
0929 00:50:03.143497 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:03.143501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:03.143560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:03.143640 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:03.145060 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.145669 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.146859 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.147089 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.147201 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:03.149880 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:03.149884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:03.149887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:03.149922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:03.151668 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.152375 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.152378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.152380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.152500 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.152956 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.152959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.152960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.153161 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:03.153246 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:03.153763 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 00:50:03.153873 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:50:03.154867 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:03.141404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:32839 (local address 127.11.40.195:55719)
0929 00:50:03.141873 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:03.141917 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:03.143185 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:03.143837 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:03.143844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:03.144243 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:03.144917 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:03.144938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.145800 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.145804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:03.146709 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:03.146718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.147243 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.147247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:03.147421 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:03.148035 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:03.148051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:03.149669 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:03.151793 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:03.151799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:03.151826 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:03.152173 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:03.152613 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:03.152616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:03.152617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:03.152769 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:03.153256 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:03.153262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:03.153513 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 00:50:03.153672 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 00:50:03.153968 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260929 00:50:03.156082 15475 heartbeater.cc:347] Connected to a master server at 127.11.40.254:32839
I20260929 00:50:03.156332 15475 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:03.156906 15475 heartbeater.cc:511] Master 127.11.40.254:32839 requested a full tablet report, sending...
I20260929 00:50:03.157981 15016 ts_manager.cc:194] Registered new tserver with Master: 18c877d121f243348a5d5b4537230d3d (127.11.40.195:35453)
I20260929 00:50:03.158602 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:55719
I20260929 00:50:03.163959 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:50:03.192593 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:03.181576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.11.40.193:39215)
0929 00:50:03.181842 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:03.181861 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:03.181890 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:50:03.181958 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:03.181963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:03.182158 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:03.182289 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:03.183063 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.183945 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.184936 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.185354 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.185453 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:03.188181 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:03.188196 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:03.188212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:03.188244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:03.190438 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.191026 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.191032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.191033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.191124 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.191450 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.191454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.191455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.191744 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:03.191999 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:03.192228 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:50:03.192411 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:03.205794 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:03.195984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.11.40.254:32839)
0929 00:50:03.196145 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:03.196152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:03.196171 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:50:03.196354 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:03.196359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:03.196430 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:03.196515 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:03.197346 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.198226 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.199386 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:03.199621 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:03.199710 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:03.202471 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:03.202475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:03.202477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:03.202513 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:03.203869 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.204364 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.204368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.204370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.204460 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:03.204754 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:03.204759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:03.204760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:03.205066 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:03.205218 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:03.205497 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 00:50:03.205626 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:50:03.206311 15016 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54864
I20260929 00:50:03.207134 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15078
I20260929 00:50:03.213135 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15212
I20260929 00:50:03.218616 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15346
I20260929 00:50:03.224197 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 14984
2026-09-29T00:50:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2290 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[15487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[15487](info): set up 2 sockets
Sep 29 00:50:03 dist-test-slave-5309 krb5kdc[15487](info): commencing operation
krb5kdc: starting...
W20260929 00:50:04.271390 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 00:50:04.459503 11427 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.188s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 00:50:04.622001 11427 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.162s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 00:50:04.755053 11427 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.133s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/test-user.keytab.
W20260929 00:50:05.424572 11427 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.669s user 0.002s sys 0.000s
Sep 29 00:50:05 dist-test-slave-5309 krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
W20260929 00:50:06.193145 11427 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.40.254: real 0.745s user 0.000s sys 0.003s
I20260929 00:50:06.193769 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34231
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:06.312238 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:06.312538 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:06.312584 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:06.316695 15500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:06.316761 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:06.316781 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:06.316797 15500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:06.316874 15500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:06.322270 15500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:06.323338 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:06.324378 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:06.332880 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:06.332935 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:06.332969 15508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:06.333261 15500 server_base.cc:1034] running on GCE node
I20260929 00:50:06.333833 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:06.335021 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:06.336210 15500 hybrid_clock.cc:648] HybridClock initialized: now 1790643006336183 us; error 38 us; skew 500 ppm
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:06.339185 15500 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:06.339936 15500 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:06.340003 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:06.340217 15500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:06.342377 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "05bc03444f9c4a8b8bb14ea8e343a157"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "74ef9dc09dc331403fd07bbee0188244"
server_key_iv: "4e1dcea106aed9b1f5912a06bc9ea5a2"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.342955 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "05bc03444f9c4a8b8bb14ea8e343a157"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "74ef9dc09dc331403fd07bbee0188244"
server_key_iv: "4e1dcea106aed9b1f5912a06bc9ea5a2"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.346808 15500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:06.349363 15514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.350621 15500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:06.350724 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "05bc03444f9c4a8b8bb14ea8e343a157"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "74ef9dc09dc331403fd07bbee0188244"
server_key_iv: "4e1dcea106aed9b1f5912a06bc9ea5a2"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.350809 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:06.366593 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:06.370035 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:06.370201 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:06.377333 15500 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34231
I20260929 00:50:06.377624 15566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34231 every 8 connection(s)
I20260929 00:50:06.378058 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:06.380877 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15500
I20260929 00:50:06.381017 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:06.381106 15567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:06.381335 11427 external_mini_cluster.cc:1428] Setting key 5ec5b7eab7e91b6a15fa5194ca32a86e
I20260929 00:50:06.389323 15567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: Bootstrap starting.
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:06.391944 15567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:06.392696 15567 log.cc:824] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:06.395190 15567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: No bootstrap required, opened a new log
I20260929 00:50:06.396425 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:06.382752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.11.40.254:34231)
0929 00:50:06.383185 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:06.383204 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:06.383251 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:50:06.383870 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:06.383886 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:06.384261 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:06.384559 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:06.385568 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.386676 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.387627 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.388110 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.388290 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:06.391196 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:06.391202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:06.391216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:06.391248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:06.393373 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.393957 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.393964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.393970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.394097 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.394393 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.394396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.394397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.394802 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:06.394961 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:06.395309 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 00:50:06.395680 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
I20260929 00:50:06.399034 15567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } }
I20260929 00:50:06.399263 15567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:06.399334 15567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc03444f9c4a8b8bb14ea8e343a157, State: Initialized, Role: FOLLOWER
I20260929 00:50:06.399752 15567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } }
I20260929 00:50:06.399919 15567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:06.399983 15567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:06.400092 15567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:06.401130 15567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } }
I20260929 00:50:06.401479 15567 leader_election.cc:302] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05bc03444f9c4a8b8bb14ea8e343a157; no voters:
I20260929 00:50:06.401873 15567 leader_election.cc:288] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:06.402000 15572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:06.402251 15572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [term 1 LEADER]: Becoming Leader. State: Replica: 05bc03444f9c4a8b8bb14ea8e343a157, State: Running, Role: LEADER
I20260929 00:50:06.402617 15572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } }
I20260929 00:50:06.403152 15567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:06.404312 15574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05bc03444f9c4a8b8bb14ea8e343a157. Latest consensus state: current_term: 1 leader_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } } }
I20260929 00:50:06.404458 15574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:06.404727 15573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc03444f9c4a8b8bb14ea8e343a157" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34231 } } }
I20260929 00:50:06.404857 15573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157 [sys.catalog]: This master's current role is: LEADER
W20260929 00:50:06.407754 15587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:50:06.407829 15587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:50:06.407912 15581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:06.408696 15581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:06.414779 15581 catalog_manager.cc:1347] Generated new cluster ID: fa123a857ad54ca5a64c2dd5808bab35
I20260929 00:50:06.414865 15581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:06.432348 15581 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:06.433475 15581 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:06.444219 15581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: Generated new TSK 0
I20260929 00:50:06.445097 15581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:06.482275 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:06.629302 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:06.629630 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:06.629710 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:06.633920 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:06.634027 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:06.634133 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:06.639745 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:06.641009 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:06.642095 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:06.652204 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:06.652177 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:06.652173 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:06.652603 15593 server_base.cc:1034] running on GCE node
I20260929 00:50:06.653082 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:06.653748 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:06.654907 15593 hybrid_clock.cc:648] HybridClock initialized: now 1790643006654894 us; error 32 us; skew 500 ppm
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:06.658018 15593 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:06.658784 15593 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:06.658849 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:06.659113 15593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:06.661341 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "de263329ac5b4621a3008e7894b89720"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "c605c1e52a795d878c8405fe3fcbce24"
server_key_iv: "ff70b6798750b1abb7c0dd1714dfc390"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.661911 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "de263329ac5b4621a3008e7894b89720"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "c605c1e52a795d878c8405fe3fcbce24"
server_key_iv: "ff70b6798750b1abb7c0dd1714dfc390"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.665690 15593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:06.668249 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.669634 15593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:06.669776 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "de263329ac5b4621a3008e7894b89720"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "c605c1e52a795d878c8405fe3fcbce24"
server_key_iv: "ff70b6798750b1abb7c0dd1714dfc390"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.669903 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:06.680717 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:06.684281 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:06.684499 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:06.685175 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:06.686043 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:06.686115 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.686189 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:06.686230 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.696058 15593 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:35485
I20260929 00:50:06.696060 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:35485 every 8 connection(s)
I20260929 00:50:06.696784 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:06.700642 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15593
I20260929 00:50:06.700829 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:06.701164 11427 external_mini_cluster.cc:1428] Setting key ec2febcf005377ada6ae2fd415e1e40e
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:06.712505 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:06.698765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:56919 (local address 127.11.40.254:34231)
0929 00:50:06.698909 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:06.698913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:06.699788 (+ 875us) server_negotiation.cc:407] Connection header received
0929 00:50:06.701287 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:06.701293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:06.701365 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:06.701501 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:06.703115 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.703758 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.705198 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.705456 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.705570 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:06.708492 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:06.708495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:06.708499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:06.708533 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:06.710165 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.710728 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.710732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.710734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.710824 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.711213 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.711217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.711218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.711462 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:06.711573 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:06.712174 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 00:50:06.712319 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:50:06.713477 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:06.699137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34231 (local address 127.11.40.193:56919)
0929 00:50:06.699612 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:06.699658 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:06.701036 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:06.701645 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:06.701677 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:06.702166 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:06.702940 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:06.702958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.703931 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.703935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:06.705069 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:06.705079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.705636 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.705641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:06.705812 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:06.706455 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:06.706473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:06.708300 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:06.710297 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:06.710304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:06.710323 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:06.710615 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:06.710939 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:06.710942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:06.710944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:06.711108 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:06.711575 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:06.711583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:06.711884 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 00:50:06.712125 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 00:50:06.712525 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:06.715096 15721 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34231
I20260929 00:50:06.715389 15721 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:06.715914 15721 heartbeater.cc:511] Master 127.11.40.254:34231 requested a full tablet report, sending...
I20260929 00:50:06.717732 15531 ts_manager.cc:194] Registered new tserver with Master: de263329ac5b4621a3008e7894b89720 (127.11.40.193:35485)
I20260929 00:50:06.719060 15531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:56919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:06.733786 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:06.847322 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:06.847654 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:06.847703 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:06.852288 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:06.852378 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:06.852444 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:06.858352 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:06.859462 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:06.860464 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:06.870570 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:06.870630 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:06.870672 15726 server_base.cc:1034] running on GCE node
W20260929 00:50:06.870645 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:06.871385 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:06.872120 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:06.873322 15726 hybrid_clock.cc:648] HybridClock initialized: now 1790643006873300 us; error 36 us; skew 500 ppm
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:06.876446 15726 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:06.877354 15726 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:06.877456 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:06.877715 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:06.880065 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "a47db96167434fb7af6824da2bb964fd"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "f32574801ec218a69640a10cc9db2573"
server_key_iv: "f6860a0856c70331b6832cd1e27e5cc0"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.880708 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "a47db96167434fb7af6824da2bb964fd"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "f32574801ec218a69640a10cc9db2573"
server_key_iv: "f6860a0856c70331b6832cd1e27e5cc0"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.885013 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:06.887799 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.889297 15726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:06.889451 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "a47db96167434fb7af6824da2bb964fd"
format_stamp: "Formatted at 2026-09-29 00:50:06 on dist-test-slave-5309"
server_key: "f32574801ec218a69640a10cc9db2573"
server_key_iv: "f6860a0856c70331b6832cd1e27e5cc0"
server_key_version: "encryptionkey@0"
I20260929 00:50:06.889590 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:06.905588 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:06.909308 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:06.909539 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:06.910163 15726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:06.911055 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:06.911135 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.911227 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:06.911279 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:06.921334 15726 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:34937
I20260929 00:50:06.921420 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:34937 every 8 connection(s)
I20260929 00:50:06.922109 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:06.930949 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15726
I20260929 00:50:06.931105 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:06.931392 11427 external_mini_cluster.cc:1428] Setting key d90f5eaa34e8328cbc6a8b26e3f10f59
Sep 29 00:50:06 dist-test-slave-5309 krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:06.938722 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:06.924467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:47053 (local address 127.11.40.254:34231)
0929 00:50:06.924619 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:06.924624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:06.925526 (+ 902us) server_negotiation.cc:407] Connection header received
0929 00:50:06.926914 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:06.926918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:06.926984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:06.927094 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:06.928355 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.928978 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.930345 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.930584 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.930708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:06.933599 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:06.933603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:06.933609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:06.933640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:06.935616 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.936491 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.936494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.936496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.936595 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:06.937096 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:06.937099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:06.937100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:06.937309 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:06.937400 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:06.938277 (+ 877us) server_negotiation.cc:299] Negotiation successful
0929 00:50:06.938453 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:50:06.939534 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:06.924855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34231 (local address 127.11.40.194:47053)
0929 00:50:06.925357 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:06.925404 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:06.926661 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:06.927285 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:06.927292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:06.927688 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:06.928190 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:06.928205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.929129 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.929133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:06.930190 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:06.930201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:06.930751 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:06.930765 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:06.930941 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:06.931663 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:06.931681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:06.933413 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:06.935843 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:06.935859 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:06.935880 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:06.936350 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:06.936707 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:06.936713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:06.936718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:06.936982 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:06.937437 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:06.937448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:06.937861 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 00:50:06.938118 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 00:50:06.938525 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260929 00:50:06.941237 15854 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34231
I20260929 00:50:06.941504 15854 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:06.942018 15854 heartbeater.cc:511] Master 127.11.40.254:34231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:06.943162 15531 ts_manager.cc:194] Registered new tserver with Master: a47db96167434fb7af6824da2bb964fd (127.11.40.194:34937)
I20260929 00:50:06.943910 15531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:47053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:06.965262 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:07.078606 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:07.078929 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:07.078984 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:07.083139 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:07.083206 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:07.083308 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:07.089111 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:07.090268 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:07.091316 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:07.101346 15864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.101641 15867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.101577 15865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:07.101629 15859 server_base.cc:1034] running on GCE node
I20260929 00:50:07.102214 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:07.102960 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:07.104169 15859 hybrid_clock.cc:648] HybridClock initialized: now 1790643007104150 us; error 31 us; skew 500 ppm
Sep 29 00:50:07 dist-test-slave-5309 krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:07.107688 15859 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:07.108428 15859 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:07.108492 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:07.108713 15859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:07.110853 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "a14d5431c2b64798a039a11e3a259c98"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "f7412b282b721649a39a4881f650c95c"
server_key_iv: "f78e01070b678f72d2c36f96dd29fc58"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.111472 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "a14d5431c2b64798a039a11e3a259c98"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "f7412b282b721649a39a4881f650c95c"
server_key_iv: "f78e01070b678f72d2c36f96dd29fc58"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.115357 15859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 00:50:07.117908 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:07.119246 15859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:07.119347 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "a14d5431c2b64798a039a11e3a259c98"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "f7412b282b721649a39a4881f650c95c"
server_key_iv: "f78e01070b678f72d2c36f96dd29fc58"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.119495 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:07.169555 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:07.173265 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:07.173528 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:07.174180 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:07.175050 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:07.175099 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:07.175140 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:07.175156 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:07.185792 15859 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:35877
I20260929 00:50:07.185808 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:35877 every 8 connection(s)
I20260929 00:50:07.186467 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:07.192705 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 15859
I20260929 00:50:07.192870 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:07.193243 11427 external_mini_cluster.cc:1428] Setting key dd6b010201583c6389b062abdc7ae376
Sep 29 00:50:07 dist-test-slave-5309 krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:07.200850 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.188354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:43709 (local address 127.11.40.254:34231)
0929 00:50:07.188503 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.188506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.189398 (+ 892us) server_negotiation.cc:407] Connection header received
0929 00:50:07.190717 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.190721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.190807 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.190904 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:07.192113 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.192796 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.194434 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.194745 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.194853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.197464 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:07.197467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:07.197469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:07.197493 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:07.198873 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:07.199337 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:07.199339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:07.199341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:07.199454 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:07.199767 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:07.199770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:07.199772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:07.200002 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:07.200082 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.200573 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.200681 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:50:07.201730 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.188709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34231 (local address 127.11.40.195:43709)
0929 00:50:07.189203 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.189247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.190498 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.191086 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.191093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.191472 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:07.191956 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.191970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.192983 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.192987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.194218 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.194233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.194920 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.194924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.195071 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.195750 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:07.195767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:07.197269 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:07.198981 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:07.198986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:07.198998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:07.199250 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:07.199536 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:07.199538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:07.199540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:07.199674 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:07.200098 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:07.200104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:07.200347 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.200522 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.200785 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":108,"threads_started":1}
I20260929 00:50:07.202896 15987 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34231
I20260929 00:50:07.203189 15987 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:07.203614 15987 heartbeater.cc:511] Master 127.11.40.254:34231 requested a full tablet report, sending...
I20260929 00:50:07.204694 15531 ts_manager.cc:194] Registered new tserver with Master: a14d5431c2b64798a039a11e3a259c98 (127.11.40.195:35877)
I20260929 00:50:07.205384 15531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:43709
I20260929 00:50:07.207281 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:50:07.217374 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.209352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.11.40.254:34231)
0929 00:50:07.209529 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.209533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.209672 (+ 139us) server_negotiation.cc:407] Connection header received
0929 00:50:07.209764 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.209766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.209803 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.209882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:07.210571 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.211067 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.212223 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.212458 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.212568 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.214291 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:07.214295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:07.214298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:07.214330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:07.215651 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:07.216077 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:07.216081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:07.216083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:07.216174 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:07.216454 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:07.216458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:07.216460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:07.216685 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:07.216851 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.217092 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.217211 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 00:50:07.220360 15531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:07.223623 15531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:07.237748 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.232016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:34937 (local address 127.0.0.1:54078)
0929 00:50:07.232341 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.232388 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.232512 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.233353 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.233359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.233385 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:07.233529 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.233538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.234955 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.234958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.236081 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.236091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.237196 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.237200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.237308 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.237348 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.237467 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.237587 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260929 00:50:07.237819 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.232160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.11.40.194:34937)
0929 00:50:07.232485 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.232495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.232540 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:50:07.232689 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.232694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.232946 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.233141 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:07.233645 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.234814 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.236225 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.236862 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.236977 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.237135 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.237536 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.237656 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
I20260929 00:50:07.238606 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.232978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.11.40.193:35485)
0929 00:50:07.233529 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.233535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.233686 (+ 151us) server_negotiation.cc:407] Connection header received
0929 00:50:07.233860 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.233869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.234078 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.234166 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:07.234559 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.235469 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.236989 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.237596 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.237695 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.237833 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.238390 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.238492 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":128,"threads_started":1}
I20260929 00:50:07.239449 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.233132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:35877 (local address 127.0.0.1:57634)
0929 00:50:07.233836 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.233854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.233927 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.234366 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.234369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.234393 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:07.234462 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.234467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.236932 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.236939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.238036 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.238046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.239054 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.239061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.239154 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.239177 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.239259 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.239333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"spinlock_wait_cycles":3072,"thread_start_us":93,"threads_started":1}
I20260929 00:50:07.239586 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.233562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.11.40.195:35877)
0929 00:50:07.233952 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.233963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.233986 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:50:07.234054 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.234059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.234238 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.234385 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:07.235613 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.236772 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.238155 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.238924 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.239035 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.239168 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.239308 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.239420 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":97,"threads_started":1}
I20260929 00:50:07.239048 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.232774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:35485 (local address 127.0.0.1:56984)
0929 00:50:07.233578 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.233594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.233699 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.234308 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.234310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.234349 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:07.234461 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.234467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.235571 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.235576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.236797 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.236835 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.237900 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.237904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.238220 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.238248 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.238827 (+ 579us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.238910 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"spinlock_wait_cycles":2944,"thread_start_us":123,"threads_started":1}
I20260929 00:50:07.241663 15788 tablet_service.cc:1467] Processing CreateTablet for tablet b18103a65cf54c299a18e754ebdccb25 (DEFAULT_TABLE table=test-table [id=4ab4d969ea934fd9bdac3afc26e860e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:07.242261 15921 tablet_service.cc:1467] Processing CreateTablet for tablet b18103a65cf54c299a18e754ebdccb25 (DEFAULT_TABLE table=test-table [id=4ab4d969ea934fd9bdac3afc26e860e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:07.242887 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18103a65cf54c299a18e754ebdccb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:07.242736 15655 tablet_service.cc:1467] Processing CreateTablet for tablet b18103a65cf54c299a18e754ebdccb25 (DEFAULT_TABLE table=test-table [id=4ab4d969ea934fd9bdac3afc26e860e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:07.243301 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18103a65cf54c299a18e754ebdccb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:07.243676 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18103a65cf54c299a18e754ebdccb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:07.250692 16004 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Bootstrap starting.
I20260929 00:50:07.250850 16005 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Bootstrap starting.
I20260929 00:50:07.253150 16006 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Bootstrap starting.
I20260929 00:50:07.253228 16005 tablet_bootstrap.cc:654] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:07.253228 16004 tablet_bootstrap.cc:654] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:07.254235 16005 log.cc:824] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:07.254541 16004 log.cc:824] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:07.255307 16006 tablet_bootstrap.cc:654] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:07.256547 16004 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: No bootstrap required, opened a new log
I20260929 00:50:07.256565 16005 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: No bootstrap required, opened a new log
I20260929 00:50:07.256574 16006 log.cc:824] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:07.256752 16005 ts_tablet_manager.cc:1397] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:50:07.256752 16004 ts_tablet_manager.cc:1397] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 00:50:07.258385 16006 tablet_bootstrap.cc:492] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: No bootstrap required, opened a new log
I20260929 00:50:07.258591 16006 ts_tablet_manager.cc:1397] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:50:07.260942 16004 raft_consensus.cc:348] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.261260 16004 raft_consensus.cc:374] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:07.261353 16004 raft_consensus.cc:729] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14d5431c2b64798a039a11e3a259c98, State: Initialized, Role: FOLLOWER
I20260929 00:50:07.261387 16005 raft_consensus.cc:348] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.261518 16006 raft_consensus.cc:348] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.261723 16005 raft_consensus.cc:374] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:07.261788 16005 raft_consensus.cc:729] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de263329ac5b4621a3008e7894b89720, State: Initialized, Role: FOLLOWER
I20260929 00:50:07.261806 16006 raft_consensus.cc:374] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:07.261860 16006 raft_consensus.cc:729] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47db96167434fb7af6824da2bb964fd, State: Initialized, Role: FOLLOWER
I20260929 00:50:07.261862 16004 consensus_queue.cc:260] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.262328 16005 consensus_queue.cc:260] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.262336 16006 consensus_queue.cc:260] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.262595 15987 heartbeater.cc:503] Master 127.11.40.254:34231 was elected leader, sending a full tablet report...
I20260929 00:50:07.263410 16006 ts_tablet_manager.cc:1428] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:50:07.263423 16005 ts_tablet_manager.cc:1428] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:50:07.263731 15721 heartbeater.cc:503] Master 127.11.40.254:34231 was elected leader, sending a full tablet report...
I20260929 00:50:07.263736 15854 heartbeater.cc:503] Master 127.11.40.254:34231 was elected leader, sending a full tablet report...
I20260929 00:50:07.264312 16004 ts_tablet_manager.cc:1428] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 00:50:07.423343 15855 tablet.cc:2369] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:07.437433 15988 tablet.cc:2369] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:07.448175 15722 tablet.cc:2369] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:07.493425 16012 raft_consensus.cc:482] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:07.493695 16012 raft_consensus.cc:504] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.495014 16012 leader_election.cc:288] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47db96167434fb7af6824da2bb964fd (127.11.40.194:34937), a14d5431c2b64798a039a11e3a259c98 (127.11.40.195:35877)
I20260929 00:50:07.499593 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.495467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:38999 (local address 127.11.40.194:34937)
0929 00:50:07.495615 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.495619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.495866 (+ 247us) server_negotiation.cc:407] Connection header received
0929 00:50:07.495945 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.495950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.496038 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.496145 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:07.496440 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.497181 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.498562 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.499005 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.499096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.499137 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.499375 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.499448 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:50:07.499655 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.495376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:34937 (local address 127.11.40.193:38999)
0929 00:50:07.495695 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.495712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.495835 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.496191 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.496193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.496211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:07.496318 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.496323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.497288 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.497292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.498429 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.498438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.499130 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.499134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.499241 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.499268 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.499342 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.499471 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":18,"thread_start_us":123,"threads_started":1}
I20260929 00:50:07.499742 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.496050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:42931 (local address 127.11.40.195:35877)
0929 00:50:07.496175 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.496180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.496439 (+ 259us) server_negotiation.cc:407] Connection header received
0929 00:50:07.496573 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.496575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.496614 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.496673 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:07.496992 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.497618 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.498679 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.499141 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.499256 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.499303 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.499540 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.499610 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:50:07.499852 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.495837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:35877 (local address 127.11.40.193:42931)
0929 00:50:07.496295 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.496308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:07.496475 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:07.496708 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:07.496711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:07.496728 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:07.496879 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.496886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.497729 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.497731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.498564 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:07.498574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.499266 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.499269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:07.499395 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.499433 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:50:07.499507 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:50:07.499572 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260929 00:50:07.500437 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18103a65cf54c299a18e754ebdccb25" candidate_uuid: "de263329ac5b4621a3008e7894b89720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47db96167434fb7af6824da2bb964fd" is_pre_election: true
I20260929 00:50:07.500438 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18103a65cf54c299a18e754ebdccb25" candidate_uuid: "de263329ac5b4621a3008e7894b89720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14d5431c2b64798a039a11e3a259c98" is_pre_election: true
I20260929 00:50:07.500743 15941 raft_consensus.cc:2454] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de263329ac5b4621a3008e7894b89720 in term 0.
I20260929 00:50:07.500751 15806 raft_consensus.cc:2454] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de263329ac5b4621a3008e7894b89720 in term 0.
I20260929 00:50:07.501255 15608 leader_election.cc:302] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a14d5431c2b64798a039a11e3a259c98, de263329ac5b4621a3008e7894b89720; no voters:
I20260929 00:50:07.501534 16012 raft_consensus.cc:2781] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:07.501613 16012 raft_consensus.cc:482] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:07.501643 16012 raft_consensus.cc:3037] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:07.502786 16012 raft_consensus.cc:504] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.503139 16012 leader_election.cc:288] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [CANDIDATE]: Term 1 election: Requested vote from peers a47db96167434fb7af6824da2bb964fd (127.11.40.194:34937), a14d5431c2b64798a039a11e3a259c98 (127.11.40.195:35877)
I20260929 00:50:07.503567 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18103a65cf54c299a18e754ebdccb25" candidate_uuid: "de263329ac5b4621a3008e7894b89720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47db96167434fb7af6824da2bb964fd"
I20260929 00:50:07.503587 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18103a65cf54c299a18e754ebdccb25" candidate_uuid: "de263329ac5b4621a3008e7894b89720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14d5431c2b64798a039a11e3a259c98"
I20260929 00:50:07.503713 15806 raft_consensus.cc:3037] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:07.503724 15941 raft_consensus.cc:3037] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:07.504931 15806 raft_consensus.cc:2454] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de263329ac5b4621a3008e7894b89720 in term 1.
I20260929 00:50:07.505000 15941 raft_consensus.cc:2454] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de263329ac5b4621a3008e7894b89720 in term 1.
I20260929 00:50:07.505228 15608 leader_election.cc:302] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47db96167434fb7af6824da2bb964fd, de263329ac5b4621a3008e7894b89720; no voters:
I20260929 00:50:07.505407 16012 raft_consensus.cc:2781] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:07.505643 16012 raft_consensus.cc:686] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [term 1 LEADER]: Becoming Leader. State: Replica: de263329ac5b4621a3008e7894b89720, State: Running, Role: LEADER
I20260929 00:50:07.505996 16012 consensus_queue.cc:237] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } }
I20260929 00:50:07.509230 15530 catalog_manager.cc:5495] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 reported cstate change: term changed from 0 to 1, leader changed from <none> to de263329ac5b4621a3008e7894b89720 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "de263329ac5b4621a3008e7894b89720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de263329ac5b4621a3008e7894b89720" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 } health_report { overall_health: UNKNOWN } } }
I20260929 00:50:07.533659 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.530148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.11.40.193:35485)
0929 00:50:07.530283 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.530286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.530299 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:07.530377 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.530379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.530433 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.530524 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:07.530835 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.531354 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.532249 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:07.532478 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:07.532563 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:07.532900 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:07.533338 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:07.533436 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.533496 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.533545 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:07.539750 15941 raft_consensus.cc:1261] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 1 FOLLOWER]: Refusing update from remote peer de263329ac5b4621a3008e7894b89720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:07.539750 15806 raft_consensus.cc:1261] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 1 FOLLOWER]: Refusing update from remote peer de263329ac5b4621a3008e7894b89720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:07.540509 16012 consensus_queue.cc:1035] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47db96167434fb7af6824da2bb964fd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:07.540700 16015 consensus_queue.cc:1035] T b18103a65cf54c299a18e754ebdccb25 P de263329ac5b4621a3008e7894b89720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14d5431c2b64798a039a11e3a259c98" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:07.559355 15529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:07.559567 15529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:07.562328 15529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05bc03444f9c4a8b8bb14ea8e343a157: Sending DeleteTablet for 3 replicas of tablet b18103a65cf54c299a18e754ebdccb25
I20260929 00:50:07.563333 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet b18103a65cf54c299a18e754ebdccb25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:07 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:57634
I20260929 00:50:07.563340 15788 tablet_service.cc:1514] Processing DeleteTablet for tablet b18103a65cf54c299a18e754ebdccb25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:07 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:54078
I20260929 00:50:07.564024 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15593
I20260929 00:50:07.564734 16028 tablet_replica.cc:330] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: stopping tablet replica
I20260929 00:50:07.565132 16028 raft_consensus.cc:2229] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:07.565471 16028 raft_consensus.cc:2258] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:07.565871 16029 tablet_replica.cc:330] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: stopping tablet replica
I20260929 00:50:07.566221 16029 raft_consensus.cc:2229] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:07.566577 16029 raft_consensus.cc:2258] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:07.567754 16028 ts_tablet_manager.cc:1905] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:07.568068 16029 ts_tablet_manager.cc:1905] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:07.570725 16029 ts_tablet_manager.cc:1918] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:07.570726 16028 ts_tablet_manager.cc:1918] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:07.570837 16029 log.cc:1196] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/b18103a65cf54c299a18e754ebdccb25
I20260929 00:50:07.570837 16028 log.cc:1196] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/b18103a65cf54c299a18e754ebdccb25
I20260929 00:50:07.571174 16028 ts_tablet_manager.cc:1939] T b18103a65cf54c299a18e754ebdccb25 P a14d5431c2b64798a039a11e3a259c98: Deleting consensus metadata
I20260929 00:50:07.571174 16029 ts_tablet_manager.cc:1939] T b18103a65cf54c299a18e754ebdccb25 P a47db96167434fb7af6824da2bb964fd: Deleting consensus metadata
W20260929 00:50:07.571720 15515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:35485 (error 108)
I20260929 00:50:07.572677 15515 catalog_manager.cc:4841] TS a14d5431c2b64798a039a11e3a259c98 (127.11.40.195:35877): tablet b18103a65cf54c299a18e754ebdccb25 (table test-table [id=4ab4d969ea934fd9bdac3afc26e860e7]) successfully deleted
I20260929 00:50:07.572794 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.572494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:35485 (local address 127.0.0.1:57014)
0929 00:50:07.572626 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:07.572702 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 00:50:07.572897 15515 catalog_manager.cc:4841] TS a47db96167434fb7af6824da2bb964fd (127.11.40.194:34937): tablet b18103a65cf54c299a18e754ebdccb25 (table test-table [id=4ab4d969ea934fd9bdac3afc26e860e7]) successfully deleted
W20260929 00:50:07.573055 15515 catalog_manager.cc:4584] TS de263329ac5b4621a3008e7894b89720 (127.11.40.193:35485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b18103a65cf54c299a18e754ebdccb25: Network error: Client connection negotiation failed: client connection to 127.11.40.193:35485: connect: Connection refused (error 111)
I20260929 00:50:07.573227 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15726
I20260929 00:50:07.580319 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15859
I20260929 00:50:07.588657 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 15500
2026-09-29T00:50:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4365 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T00:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:07Z Disabled control of system clock
I20260929 00:50:07.619969 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39087
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:50:07.734460 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:07.734763 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:07.734803 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:07.739074 16034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:07.739143 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:07.739159 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:07.739176 16034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:07.739192 16034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:07.744023 16034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:39087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:07.745150 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:07.746616 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:07.755404 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.755498 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.755420 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:07.755654 16034 server_base.cc:1034] running on GCE node
I20260929 00:50:07.756331 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:07.757678 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:07.758906 16034 hybrid_clock.cc:648] HybridClock initialized: now 1790643007758865 us; error 55 us; skew 500 ppm
I20260929 00:50:07.761047 16034 webserver.cc:460] Webserver started at http://127.11.40.254:33805/ using document root <none> and password file <none>
I20260929 00:50:07.761648 16034 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:07.761749 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:07.761997 16034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:07.764135 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "38f207b8e98f44b1abfef7bdcb8033e4"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "2ebf62eea3eac7c76ea822198e41e247"
server_key_iv: "59dacbcb787af8d9388003a4390a9f72"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.764722 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "38f207b8e98f44b1abfef7bdcb8033e4"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "2ebf62eea3eac7c76ea822198e41e247"
server_key_iv: "59dacbcb787af8d9388003a4390a9f72"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.768673 16034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:07.771140 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:07.772517 16034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:07.772679 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "38f207b8e98f44b1abfef7bdcb8033e4"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "2ebf62eea3eac7c76ea822198e41e247"
server_key_iv: "59dacbcb787af8d9388003a4390a9f72"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.772842 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:07.777532 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:07.778149 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:07.778337 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:07.785205 16034 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:39087
I20260929 00:50:07.785249 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:39087 every 8 connection(s)
I20260929 00:50:07.786268 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:07.789610 16101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:07.796329 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16034
I20260929 00:50:07.796475 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:07.796720 16101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: Bootstrap starting.
I20260929 00:50:07.796787 11427 external_mini_cluster.cc:1428] Setting key 049548c489c0eded44820833a46bc86d
I20260929 00:50:07.800009 16101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:07.801077 16101 log.cc:824] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:07.803704 16101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: No bootstrap required, opened a new log
I20260929 00:50:07.806810 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:07.800144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.11.40.254:39087)
0929 00:50:07.801054 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:07.801074 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:07.801138 (+ 64us) server_negotiation.cc:407] Connection header received
0929 00:50:07.802141 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:07.802166 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:07.802645 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:07.803097 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:07.804858 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:07.804864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:07.804880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:07.804894 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:07.804990 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:07.805017 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:07.805133 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:07.805620 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 00:50:07.805908 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":159,"threads_started":1}
I20260929 00:50:07.808338 16101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } }
I20260929 00:50:07.808571 16101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:07.808633 16101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f207b8e98f44b1abfef7bdcb8033e4, State: Initialized, Role: FOLLOWER
I20260929 00:50:07.809167 16101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } }
I20260929 00:50:07.809351 16101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:07.809420 16101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:07.809527 16101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:07.810469 16101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } }
I20260929 00:50:07.810819 16101 leader_election.cc:302] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f207b8e98f44b1abfef7bdcb8033e4; no voters:
I20260929 00:50:07.811165 16101 leader_election.cc:288] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:07.811247 16106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:07.811419 16106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [term 1 LEADER]: Becoming Leader. State: Replica: 38f207b8e98f44b1abfef7bdcb8033e4, State: Running, Role: LEADER
I20260929 00:50:07.811710 16106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } }
I20260929 00:50:07.812454 16101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:07.813416 16108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f207b8e98f44b1abfef7bdcb8033e4. Latest consensus state: current_term: 1 leader_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } } }
I20260929 00:50:07.813459 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f207b8e98f44b1abfef7bdcb8033e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39087 } } }
I20260929 00:50:07.813539 16108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:07.813573 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:07.813920 16115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:07.817119 16115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:07.822602 16115 catalog_manager.cc:1347] Generated new cluster ID: 31398786d12a48a485c45b99006f1ae5
I20260929 00:50:07.822671 16115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:07.828923 16115 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:07.829294 16115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:07.835739 16115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: Generated new TSK 0
I20260929 00:50:07.836460 16115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:07.838958 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--builtin_ntp_servers=127.11.40.212:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:50:07.956243 16125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:07.956691 16125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:07.956784 16125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:07.961118 16125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:07.961189 16125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:07.961311 16125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:07.966406 16125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:07.967589 16125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:07.969198 16125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:07.980060 16133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.980096 16131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:07.980064 16130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:07.980495 16125 server_base.cc:1034] running on GCE node
I20260929 00:50:07.981005 16125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:07.981698 16125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:07.982887 16125 hybrid_clock.cc:648] HybridClock initialized: now 1790643007982867 us; error 33 us; skew 500 ppm
I20260929 00:50:07.985168 16125 webserver.cc:460] Webserver started at http://127.11.40.193:45923/ using document root <none> and password file <none>
I20260929 00:50:07.985814 16125 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:07.985882 16125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:07.986120 16125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:07.988407 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "8b757c0ec12c4d96bc847d279193825f"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "79b6cbbf1fe00c86ff4c78dd74de8313"
server_key_iv: "5b44bea96b19ff37dd0fe347f3c34341"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.989066 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "8b757c0ec12c4d96bc847d279193825f"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "79b6cbbf1fe00c86ff4c78dd74de8313"
server_key_iv: "5b44bea96b19ff37dd0fe347f3c34341"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.993047 16125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:07.995805 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:07.997179 16125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:07.997309 16125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "8b757c0ec12c4d96bc847d279193825f"
format_stamp: "Formatted at 2026-09-29 00:50:07 on dist-test-slave-5309"
server_key: "79b6cbbf1fe00c86ff4c78dd74de8313"
server_key_iv: "5b44bea96b19ff37dd0fe347f3c34341"
server_key_version: "encryptionkey@0"
I20260929 00:50:07.997431 16125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:08.002211 16125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:08.002862 16125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:08.003055 16125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:08.003626 16125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:08.004604 16125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:08.004652 16125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.004691 16125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:08.004732 16125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.013986 16125 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45681
I20260929 00:50:08.014079 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45681 every 8 connection(s)
I20260929 00:50:08.015077 16125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:08.016024 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16125
I20260929 00:50:08.016148 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:08.016434 11427 external_mini_cluster.cc:1428] Setting key 539ce19535ca26acd56652f75ef4a939
I20260929 00:50:08.018743 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--builtin_ntp_servers=127.11.40.212:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 00:50:08.022830 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.017224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:40083 (local address 127.11.40.254:39087)
0929 00:50:08.017377 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.017382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.018873 (+ 1491us) server_negotiation.cc:407] Connection header received
0929 00:50:08.020950 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.020955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.021009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.021109 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.022283 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.022289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.022295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.022313 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.022339 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.022352 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.022461 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.022589 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.022645 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:50:08.023767 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.017691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39087 (local address 127.11.40.193:40083)
0929 00:50:08.018666 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.018728 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.020677 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.021246 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.021253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.021361 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.022002 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.022003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.022047 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.022119 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.022444 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.022450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.022455 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.022596 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.022863 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":152,"threads_started":1}
I20260929 00:50:08.025096 16253 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39087
I20260929 00:50:08.025336 16253 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:08.025758 16253 heartbeater.cc:511] Master 127.11.40.254:39087 requested a full tablet report, sending...
I20260929 00:50:08.027289 16065 ts_manager.cc:194] Registered new tserver with Master: 8b757c0ec12c4d96bc847d279193825f (127.11.40.193:45681)
W20260929 00:50:08.134421 16256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:08.134694 16256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:08.134734 16256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:08.138859 16256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:08.138940 16256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:08.138999 16256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:08.144273 16256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:08.145627 16256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:08.147101 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:08.157155 16261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:08.157150 16264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:08.157254 16256 server_base.cc:1034] running on GCE node
W20260929 00:50:08.157191 16262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:08.158048 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:08.158704 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:08.159868 16256 hybrid_clock.cc:648] HybridClock initialized: now 1790643008159829 us; error 52 us; skew 500 ppm
I20260929 00:50:08.161881 16256 webserver.cc:460] Webserver started at http://127.11.40.194:44241/ using document root <none> and password file <none>
I20260929 00:50:08.162479 16256 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:08.162544 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:08.162786 16256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:08.164971 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "a03caee7bb86480eac81b30f90c071b6"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "8c2c4c705cb9e028629988377809bc69"
server_key_iv: "0d94cf1252a19deeade88c352077e3be"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.165555 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "a03caee7bb86480eac81b30f90c071b6"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "8c2c4c705cb9e028629988377809bc69"
server_key_iv: "0d94cf1252a19deeade88c352077e3be"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.169339 16256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:08.171973 16270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.173357 16256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:08.173506 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "a03caee7bb86480eac81b30f90c071b6"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "8c2c4c705cb9e028629988377809bc69"
server_key_iv: "0d94cf1252a19deeade88c352077e3be"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.173621 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:08.178453 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:08.179049 16256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:08.179230 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:08.179770 16256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:08.180662 16256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:08.180708 16256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.180774 16256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:08.180848 16256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.189555 16256 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:45785
I20260929 00:50:08.189585 16383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:45785 every 8 connection(s)
I20260929 00:50:08.190436 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:08.196079 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16256
I20260929 00:50:08.196240 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:08.196554 11427 external_mini_cluster.cc:1428] Setting key a606665a7693ca0248b3a21d52239643
I20260929 00:50:08.196699 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.192214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56767 (local address 127.11.40.254:39087)
0929 00:50:08.192357 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.192360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.193487 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 00:50:08.194803 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.194806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.194853 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.194976 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.196150 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.196154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.196156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.196170 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.196215 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.196227 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.196319 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.196479 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.196561 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:08.197697 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.192619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39087 (local address 127.11.40.194:56767)
0929 00:50:08.193308 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.193348 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.194586 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.195143 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.195149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.195238 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.195866 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.195868 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.195912 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.195983 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.196307 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.196315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.196321 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.196534 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.196776 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":183,"threads_started":1}
I20260929 00:50:08.198778 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--builtin_ntp_servers=127.11.40.212:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 00:50:08.198843 16384 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39087
I20260929 00:50:08.199080 16384 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:08.199615 16384 heartbeater.cc:511] Master 127.11.40.254:39087 requested a full tablet report, sending...
I20260929 00:50:08.201006 16065 ts_manager.cc:194] Registered new tserver with Master: a03caee7bb86480eac81b30f90c071b6 (127.11.40.194:45785)
W20260929 00:50:08.316334 16387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:08.316645 16387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:08.316732 16387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:08.320959 16387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:08.321058 16387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:08.321153 16387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:08.326711 16387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:08.328369 16387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:08.329906 16387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:08.339973 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:08.339973 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:08.339974 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:08.340029 16387 server_base.cc:1034] running on GCE node
I20260929 00:50:08.340760 16387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:08.341481 16387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:08.342758 16387 hybrid_clock.cc:648] HybridClock initialized: now 1790643008342725 us; error 51 us; skew 500 ppm
I20260929 00:50:08.345108 16387 webserver.cc:460] Webserver started at http://127.11.40.195:38207/ using document root <none> and password file <none>
I20260929 00:50:08.345769 16387 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:08.345872 16387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:08.346122 16387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:08.348424 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "b54b59d18e4b4889afb1758fbb796cf5"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "f55a333a6b8a4f3b8818c731140264de"
server_key_iv: "72a303d13ac572714566fe47d8f99738"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.349090 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "b54b59d18e4b4889afb1758fbb796cf5"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "f55a333a6b8a4f3b8818c731140264de"
server_key_iv: "72a303d13ac572714566fe47d8f99738"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.353258 16387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:08.355973 16401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.357398 16387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:08.357571 16387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "b54b59d18e4b4889afb1758fbb796cf5"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "f55a333a6b8a4f3b8818c731140264de"
server_key_iv: "72a303d13ac572714566fe47d8f99738"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.357697 16387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:08.362514 16387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:08.363173 16387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:08.363368 16387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:08.363991 16387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:08.365015 16387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:08.365083 16387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.365149 16387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:08.365199 16387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.374157 16387 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:45113
I20260929 00:50:08.374187 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:45113 every 8 connection(s)
I20260929 00:50:08.375121 16387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:08.376720 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16387
I20260929 00:50:08.376891 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:08.377259 11427 external_mini_cluster.cc:1428] Setting key df70191041a06511a232ed1b3e284ef4
I20260929 00:50:08.382290 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.377328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:49353 (local address 127.11.40.254:39087)
0929 00:50:08.377473 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.377479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.378970 (+ 1491us) server_negotiation.cc:407] Connection header received
0929 00:50:08.380450 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.380453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.380498 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.380563 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.381808 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.381811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.381814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.381827 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.381848 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.381856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.381970 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.382108 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.382155 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:50:08.383180 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.377990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39087 (local address 127.11.40.195:49353)
0929 00:50:08.378831 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.378874 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.380241 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.380739 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.380746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.380901 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.381546 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.381548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.381592 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.381663 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.381953 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.381961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.381967 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.382155 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.382405 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260929 00:50:08.384328 16515 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39087
I20260929 00:50:08.384544 16515 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:08.384979 16515 heartbeater.cc:511] Master 127.11.40.254:39087 requested a full tablet report, sending...
I20260929 00:50:08.386049 16065 ts_manager.cc:194] Registered new tserver with Master: b54b59d18e4b4889afb1758fbb796cf5 (127.11.40.195:45113)
I20260929 00:50:08.391366 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:50:08.394884 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.393452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.11.40.254:39087)
0929 00:50:08.393590 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.393594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.393799 (+ 205us) server_negotiation.cc:407] Connection header received
0929 00:50:08.393893 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.393895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.393928 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.394001 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.394598 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.394601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.394603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.394613 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.394631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.394638 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.394705 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.394745 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.394778 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:50:08.397758 16065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:08.401070 16065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:08.414541 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.410776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45113 (local address 127.0.0.1:41508)
0929 00:50:08.411954 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.412009 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.412173 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.412888 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.412891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.412903 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.413638 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.413640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.413663 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.413736 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.414127 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.414136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.414142 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.414283 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.414359 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":139,"threads_started":1}
I20260929 00:50:08.414753 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.410851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.11.40.194:45785)
0929 00:50:08.411717 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.411731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.412105 (+ 374us) server_negotiation.cc:407] Connection header received
0929 00:50:08.412383 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.412396 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.412599 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.412739 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.414092 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.414100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.414115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.414128 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.414165 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.414196 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.414317 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.414498 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.414574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":117,"threads_started":1}
I20260929 00:50:08.414866 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.411734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.11.40.193:45681)
0929 00:50:08.412196 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.412210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.412538 (+ 328us) server_negotiation.cc:407] Connection header received
0929 00:50:08.412602 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.412607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.412748 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.412967 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.414159 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.414165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.414176 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.414190 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.414238 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.414276 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.414377 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.414602 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.414682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":107,"threads_started":1}
I20260929 00:50:08.415306 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.411209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.11.40.195:45113)
0929 00:50:08.411601 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.411616 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.412313 (+ 697us) server_negotiation.cc:407] Connection header received
0929 00:50:08.412385 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.412391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.412602 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.412764 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.413864 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.413873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.413887 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.413899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.413942 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.413971 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.414080 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.414306 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.415151 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":181,"threads_started":1}
I20260929 00:50:08.415385 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.410740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:45785 (local address 127.0.0.1:40586)
0929 00:50:08.411955 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.412009 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.412175 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.412768 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.412775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.412881 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.413893 (+ 1012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.413896 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.413907 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.413931 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.414359 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.414362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.414365 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.415225 (+ 860us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.415259 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":119,"threads_started":1}
I20260929 00:50:08.415040 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.411292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45681 (local address 127.0.0.1:40622)
0929 00:50:08.412423 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.412438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.412526 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.413161 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.413165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.413183 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.413985 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.413987 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.413998 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.414024 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.414421 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.414424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.414428 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.414502 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.414534 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":94,"threads_started":1}
I20260929 00:50:08.418218 16449 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3298a06454c95026633f63bdf20 (DEFAULT_TABLE table=test-table [id=0010f1345dc347cea9750e531218cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:08.418215 16318 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3298a06454c95026633f63bdf20 (DEFAULT_TABLE table=test-table [id=0010f1345dc347cea9750e531218cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:08.418215 16187 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3298a06454c95026633f63bdf20 (DEFAULT_TABLE table=test-table [id=0010f1345dc347cea9750e531218cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:08.419256 16449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3298a06454c95026633f63bdf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:08.419263 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3298a06454c95026633f63bdf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:08.419263 16318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3298a06454c95026633f63bdf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:08.426033 16532 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Bootstrap starting.
I20260929 00:50:08.427109 16533 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Bootstrap starting.
I20260929 00:50:08.428393 16532 tablet_bootstrap.cc:654] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:08.428960 16533 tablet_bootstrap.cc:654] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:08.429155 16534 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Bootstrap starting.
I20260929 00:50:08.429381 16532 log.cc:824] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:08.429805 16533 log.cc:824] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:08.431288 16534 tablet_bootstrap.cc:654] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:08.431852 16533 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: No bootstrap required, opened a new log
I20260929 00:50:08.431917 16532 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: No bootstrap required, opened a new log
I20260929 00:50:08.432118 16532 ts_tablet_manager.cc:1397] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:50:08.432118 16534 log.cc:824] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:08.432199 16533 ts_tablet_manager.cc:1397] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 00:50:08.434348 16534 tablet_bootstrap.cc:492] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: No bootstrap required, opened a new log
I20260929 00:50:08.434554 16534 ts_tablet_manager.cc:1397] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:50:08.436049 16533 raft_consensus.cc:348] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.436395 16533 raft_consensus.cc:374] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:08.436475 16533 raft_consensus.cc:729] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b757c0ec12c4d96bc847d279193825f, State: Initialized, Role: FOLLOWER
I20260929 00:50:08.436689 16532 raft_consensus.cc:348] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.437026 16532 raft_consensus.cc:374] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:08.437089 16532 raft_consensus.cc:729] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54b59d18e4b4889afb1758fbb796cf5, State: Initialized, Role: FOLLOWER
I20260929 00:50:08.437031 16533 consensus_queue.cc:260] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.437601 16532 consensus_queue.cc:260] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.437783 16253 heartbeater.cc:503] Master 127.11.40.254:39087 was elected leader, sending a full tablet report...
I20260929 00:50:08.438232 16533 ts_tablet_manager.cc:1428] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:50:08.438261 16515 heartbeater.cc:503] Master 127.11.40.254:39087 was elected leader, sending a full tablet report...
I20260929 00:50:08.438617 16532 ts_tablet_manager.cc:1428] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:50:08.440461 16534 raft_consensus.cc:348] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.440765 16534 raft_consensus.cc:374] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:08.440912 16534 raft_consensus.cc:729] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03caee7bb86480eac81b30f90c071b6, State: Initialized, Role: FOLLOWER
I20260929 00:50:08.441499 16534 consensus_queue.cc:260] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.442210 16384 heartbeater.cc:503] Master 127.11.40.254:39087 was elected leader, sending a full tablet report...
I20260929 00:50:08.442581 16534 ts_tablet_manager.cc:1428] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260929 00:50:08.516294 16254 tablet.cc:2369] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:08.616753 16538 raft_consensus.cc:482] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:08.617017 16538 raft_consensus.cc:504] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.618168 16538 leader_election.cc:288] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a03caee7bb86480eac81b30f90c071b6 (127.11.40.194:45785), b54b59d18e4b4889afb1758fbb796cf5 (127.11.40.195:45113)
I20260929 00:50:08.620525 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.618657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59883 (local address 127.11.40.194:45785)
0929 00:50:08.618820 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.618825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.618950 (+ 125us) server_negotiation.cc:407] Connection header received
0929 00:50:08.619109 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.619112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.619159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.619254 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.620016 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.620019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.620021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.620056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.620083 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.620100 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.620209 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.620339 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.620386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:50:08.620525 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.618669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:58257 (local address 127.11.40.195:45113)
0929 00:50:08.618824 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.618828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.619253 (+ 425us) server_negotiation.cc:407] Connection header received
0929 00:50:08.619397 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.619400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.619466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.619576 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.620105 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.620109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.620111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.620125 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.620146 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.620160 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.620247 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.620363 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.620397 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:08.620556 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.618536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:45785 (local address 127.11.40.193:59883)
0929 00:50:08.618829 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.618847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.618994 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.619276 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.619279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.619304 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.619830 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.619832 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.619843 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.619872 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.620223 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.620228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.620232 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.620325 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.620371 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260929 00:50:08.620556 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.618536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:45113 (local address 127.11.40.193:58257)
0929 00:50:08.619156 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.619168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:08.619283 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:08.619626 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:08.619629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:08.619640 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:08.619951 (+ 311us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:08.619953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:08.619964 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:08.619983 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:08.620238 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:08.620242 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:08.620244 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 00:50:08.620352 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:50:08.620375 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":300,"thread_start_us":92,"threads_started":1}
I20260929 00:50:08.621368 16338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3298a06454c95026633f63bdf20" candidate_uuid: "8b757c0ec12c4d96bc847d279193825f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03caee7bb86480eac81b30f90c071b6" is_pre_election: true
I20260929 00:50:08.621516 16469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3298a06454c95026633f63bdf20" candidate_uuid: "8b757c0ec12c4d96bc847d279193825f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54b59d18e4b4889afb1758fbb796cf5" is_pre_election: true
I20260929 00:50:08.621722 16338 raft_consensus.cc:2454] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b757c0ec12c4d96bc847d279193825f in term 0.
I20260929 00:50:08.621826 16469 raft_consensus.cc:2454] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b757c0ec12c4d96bc847d279193825f in term 0.
I20260929 00:50:08.622274 16140 leader_election.cc:302] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b757c0ec12c4d96bc847d279193825f, a03caee7bb86480eac81b30f90c071b6, b54b59d18e4b4889afb1758fbb796cf5; no voters:
I20260929 00:50:08.622575 16538 raft_consensus.cc:2781] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:08.622661 16538 raft_consensus.cc:482] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:08.622704 16538 raft_consensus.cc:3037] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:08.623746 16538 raft_consensus.cc:504] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.624107 16538 leader_election.cc:288] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [CANDIDATE]: Term 1 election: Requested vote from peers a03caee7bb86480eac81b30f90c071b6 (127.11.40.194:45785), b54b59d18e4b4889afb1758fbb796cf5 (127.11.40.195:45113)
I20260929 00:50:08.624418 16338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3298a06454c95026633f63bdf20" candidate_uuid: "8b757c0ec12c4d96bc847d279193825f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03caee7bb86480eac81b30f90c071b6"
I20260929 00:50:08.624516 16469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3298a06454c95026633f63bdf20" candidate_uuid: "8b757c0ec12c4d96bc847d279193825f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54b59d18e4b4889afb1758fbb796cf5"
I20260929 00:50:08.624552 16338 raft_consensus.cc:3037] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:08.624683 16469 raft_consensus.cc:3037] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:08.625617 16338 raft_consensus.cc:2454] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b757c0ec12c4d96bc847d279193825f in term 1.
I20260929 00:50:08.625869 16469 raft_consensus.cc:2454] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b757c0ec12c4d96bc847d279193825f in term 1.
I20260929 00:50:08.625941 16140 leader_election.cc:302] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b757c0ec12c4d96bc847d279193825f, a03caee7bb86480eac81b30f90c071b6; no voters:
W20260929 00:50:08.626099 16516 tablet.cc:2369] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:08.626111 16538 raft_consensus.cc:2781] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:08.626338 16538 raft_consensus.cc:686] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 1 LEADER]: Becoming Leader. State: Replica: 8b757c0ec12c4d96bc847d279193825f, State: Running, Role: LEADER
I20260929 00:50:08.626664 16538 consensus_queue.cc:237] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } }
I20260929 00:50:08.629930 16065 catalog_manager.cc:5495] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b757c0ec12c4d96bc847d279193825f (127.11.40.193). New cstate: current_term: 1 leader_uuid: "8b757c0ec12c4d96bc847d279193825f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b757c0ec12c4d96bc847d279193825f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45681 } health_report { overall_health: HEALTHY } } }
I20260929 00:50:08.655225 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.653708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.11.40.193:45681)
0929 00:50:08.653842 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.653845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.653859 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:08.653963 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.653965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.654041 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.654114 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.654841 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.654843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.654845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.654853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.654874 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.654881 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.654956 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.655069 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.655105 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:50:08.661497 16338 raft_consensus.cc:1261] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 1 FOLLOWER]: Refusing update from remote peer 8b757c0ec12c4d96bc847d279193825f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:08.661496 16469 raft_consensus.cc:1261] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 1 FOLLOWER]: Refusing update from remote peer 8b757c0ec12c4d96bc847d279193825f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:08.662267 16543 consensus_queue.cc:1035] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03caee7bb86480eac81b30f90c071b6" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:08.662468 16538 consensus_queue.cc:1035] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54b59d18e4b4889afb1758fbb796cf5" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:08.678290 16065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:08.678485 16065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:08.681025 16065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f207b8e98f44b1abfef7bdcb8033e4: Sending DeleteTablet for 3 replicas of tablet 95f6f3298a06454c95026633f63bdf20
I20260929 00:50:08.681991 16187 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f6f3298a06454c95026633f63bdf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:08 UTC) from {username='slave'} at 127.0.0.1:40622
I20260929 00:50:08.681981 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f6f3298a06454c95026633f63bdf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:08 UTC) from {username='slave'} at 127.0.0.1:41508
I20260929 00:50:08.681985 16318 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f6f3298a06454c95026633f63bdf20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:08 UTC) from {username='slave'} at 127.0.0.1:40586
I20260929 00:50:08.682761 16556 tablet_replica.cc:330] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: stopping tablet replica
I20260929 00:50:08.683024 16555 tablet_replica.cc:330] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f: stopping tablet replica
I20260929 00:50:08.683257 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16125
I20260929 00:50:08.683391 16555 raft_consensus.cc:2229] T 95f6f3298a06454c95026633f63bdf20 P 8b757c0ec12c4d96bc847d279193825f [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:08.684116 16556 raft_consensus.cc:2229] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:08.684473 16556 raft_consensus.cc:2258] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:08.684900 16557 tablet_replica.cc:330] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: stopping tablet replica
I20260929 00:50:08.685199 16557 raft_consensus.cc:2229] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:08.685516 16557 raft_consensus.cc:2258] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:08.686843 16557 ts_tablet_manager.cc:1905] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:08.686932 16556 ts_tablet_manager.cc:1905] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:08.689309 16557 ts_tablet_manager.cc:1918] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:08.689438 16557 log.cc:1196] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/95f6f3298a06454c95026633f63bdf20
I20260929 00:50:08.689764 16557 ts_tablet_manager.cc:1939] T 95f6f3298a06454c95026633f63bdf20 P a03caee7bb86480eac81b30f90c071b6: Deleting consensus metadata
I20260929 00:50:08.690778 16556 ts_tablet_manager.cc:1918] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:08.690922 16556 log.cc:1196] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/95f6f3298a06454c95026633f63bdf20
I20260929 00:50:08.690995 16049 catalog_manager.cc:4841] TS a03caee7bb86480eac81b30f90c071b6 (127.11.40.194:45785): tablet 95f6f3298a06454c95026633f63bdf20 (table test-table [id=0010f1345dc347cea9750e531218cc7d]) successfully deleted
I20260929 00:50:08.691251 16556 ts_tablet_manager.cc:1939] T 95f6f3298a06454c95026633f63bdf20 P b54b59d18e4b4889afb1758fbb796cf5: Deleting consensus metadata
W20260929 00:50:08.691892 16052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:45681 (error 108)
I20260929 00:50:08.691908 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16256
I20260929 00:50:08.693436 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.693039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45681 (local address 127.0.0.1:40648)
0929 00:50:08.693245 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:08.693325 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 00:50:08.693652 16052 catalog_manager.cc:4584] TS 8b757c0ec12c4d96bc847d279193825f (127.11.40.193:45681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95f6f3298a06454c95026633f63bdf20: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45681: connect: Connection refused (error 111)
I20260929 00:50:08.694026 16051 catalog_manager.cc:4841] TS b54b59d18e4b4889afb1758fbb796cf5 (127.11.40.195:45113): tablet 95f6f3298a06454c95026633f63bdf20 (table test-table [id=0010f1345dc347cea9750e531218cc7d]) successfully deleted
I20260929 00:50:08.698028 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16387
I20260929 00:50:08.703845 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16034
2026-09-29T00:50:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1114 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 00:50:08.726351 11427 mini_oidc.cc:152] Starting JWKS server
I20260929 00:50:08.728096 11427 webserver.cc:460] Webserver started at https://127.0.0.1:33371/ using document root <none> and password file <none>
I20260929 00:50:08.728307 11427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:50:08.728591 11427 webserver.cc:460] Webserver started at http://127.0.0.1:34653/ using document root <none> and password file <none>
2026-09-29T00:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:08Z Disabled control of system clock
I20260929 00:50:08.738066 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41305
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:41305
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/testchainca.cert with env {}
W20260929 00:50:08.850711 16564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:08.851061 16564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:08.851150 16564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:08.855614 16564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:08.855727 16564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:08.855757 16564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:08.855795 16564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:08.855834 16564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:50:08.855872 16564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:50:08.855906 16564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33371/jwks.json
I20260929 00:50:08.861567 16564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:41305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:08.862828 16564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:08.863932 16564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:08.872947 16569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:08.872956 16570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:08.873096 16564 server_base.cc:1034] running on GCE node
W20260929 00:50:08.872968 16572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:08.874111 16564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:08.875341 16564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:08.876588 16564 hybrid_clock.cc:648] HybridClock initialized: now 1790643008876550 us; error 50 us; skew 500 ppm
I20260929 00:50:08.878758 16564 webserver.cc:460] Webserver started at http://127.11.40.254:33361/ using document root <none> and password file <none>
I20260929 00:50:08.879410 16564 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:08.879513 16564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:08.879762 16564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:08.882066 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "5430b54023ad49fab4df945079bc056c"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "1cb0a3d662d8e7b4e087000d604f57e7"
server_key_iv: "fd31ad65badc9e37ad135232f301e87d"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.882692 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "5430b54023ad49fab4df945079bc056c"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "1cb0a3d662d8e7b4e087000d604f57e7"
server_key_iv: "fd31ad65badc9e37ad135232f301e87d"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.886777 16564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 00:50:08.889518 16578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:08.890857 16564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:08.891008 16564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "5430b54023ad49fab4df945079bc056c"
format_stamp: "Formatted at 2026-09-29 00:50:08 on dist-test-slave-5309"
server_key: "1cb0a3d662d8e7b4e087000d604f57e7"
server_key_iv: "fd31ad65badc9e37ad135232f301e87d"
server_key_version: "encryptionkey@0"
I20260929 00:50:08.891132 16564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:08.902458 16564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:08.903178 16564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:08.903383 16564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:08.910784 16564 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:41305
I20260929 00:50:08.911036 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:41305 every 8 connection(s)
I20260929 00:50:08.911825 16564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:08.914283 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16564
I20260929 00:50:08.914408 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:08.914698 11427 external_mini_cluster.cc:1428] Setting key 369a89fc48f2cd9ecaad2a274a657dcd
I20260929 00:50:08.915792 16631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:08.921563 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c: Bootstrap starting.
I20260929 00:50:08.924005 16631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:08.924290 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:08.916142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.11.40.254:41305)
0929 00:50:08.916888 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:08.916913 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:08.916953 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:50:08.917646 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:08.917664 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:08.918072 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:08.918411 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:08.918872 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:08.919857 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:08.920790 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:08.921327 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:08.921513 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:08.922426 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:08.922431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:08.922444 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:08.922866 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:08.922924 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:08.922944 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:08.923080 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:08.923239 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 00:50:08.923501 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
I20260929 00:50:08.924961 16631 log.cc:824] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:08.927342 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c: No bootstrap required, opened a new log
I20260929 00:50:08.927397 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:50:08.931128 16631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } }
I20260929 00:50:08.931371 16631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:08.931433 16631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5430b54023ad49fab4df945079bc056c, State: Initialized, Role: FOLLOWER
I20260929 00:50:08.931874 16631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } }
I20260929 00:50:08.932017 16631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:08.932091 16631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:08.932209 16631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:08.933241 16631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } }
I20260929 00:50:08.933584 16631 leader_election.cc:302] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5430b54023ad49fab4df945079bc056c; no voters:
I20260929 00:50:08.933916 16631 leader_election.cc:288] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:08.934054 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:08.934297 16636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [term 1 LEADER]: Becoming Leader. State: Replica: 5430b54023ad49fab4df945079bc056c, State: Running, Role: LEADER
I20260929 00:50:08.934974 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } }
I20260929 00:50:08.935218 16631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:08.936623 16638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5430b54023ad49fab4df945079bc056c. Latest consensus state: current_term: 1 leader_uuid: "5430b54023ad49fab4df945079bc056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } } }
I20260929 00:50:08.936749 16638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:08.937181 16646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:08.937140 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5430b54023ad49fab4df945079bc056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5430b54023ad49fab4df945079bc056c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41305 } } }
I20260929 00:50:08.937305 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:08.940384 16646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:08.946480 16646 catalog_manager.cc:1347] Generated new cluster ID: c85fd7c25b034aac805c220f8c073fcd
I20260929 00:50:08.946573 16646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:08.985589 16646 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:08.986660 16646 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:08.994088 16646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5430b54023ad49fab4df945079bc056c: Generated new TSK 0
I20260929 00:50:08.995195 16646 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 00:50:09.040544 16661 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 00:50:09.040673 16661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:50:09.041201 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.016941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.11.40.254:41305)
0929 00:50:09.017098 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.017103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.017325 (+ 222us) server_negotiation.cc:407] Connection header received
0929 00:50:09.017577 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.017579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.017645 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.017736 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:50:09.018098 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.018679 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.019944 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.020175 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.020261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.020685 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:50:09.040675 (+ 19990us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:50:09.041072 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:50:09.041313 16634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 00:50:09.042860 16660 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:09.016840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:41305 (local address 127.0.0.1:40140)
0929 00:50:09.017229 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:09.017246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:09.017391 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:09.017789 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:09.017792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:09.017801 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:50:09.017937 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.017945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.018803 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.018807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.019803 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.019815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.020303 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.020307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.020418 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.020477 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:50:09.041068 (+ 20591us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:50:09.041360 (+ 292us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.40.254:41305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.40.254:41305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 00:50:09.043867 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16564
2026-09-29T00:50:09Z chronyd exiting
I20260929 00:50:09.081705 11427 test_util.cc:175] -----------------------------------------------
I20260929 00:50:09.081795 11427 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790642980693194-11427-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 00:50:09.082774 11427 mini_oidc.cc:152] Starting JWKS server
I20260929 00:50:09.083595 11427 webserver.cc:460] Webserver started at https://127.0.0.1:39585/ using document root <none> and password file <none>
I20260929 00:50:09.083675 11427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:50:09.083928 11427 webserver.cc:460] Webserver started at http://127.0.0.1:44305/ using document root <none> and password file <none>
2026-09-29T00:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:09Z Disabled control of system clock
I20260929 00:50:09.092968 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33181
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33181
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39585/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/testchainca.cert with env {}
W20260929 00:50:09.204746 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:09.205062 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:09.205103 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:09.209252 16667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:09.209316 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:09.209331 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:09.209349 16667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:09.209365 16667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:50:09.209381 16667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:50:09.209395 16667 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39585/jwks.json
I20260929 00:50:09.215119 16667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39585/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:09.216284 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:09.217430 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:09.227851 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:09.227949 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:09.227924 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:09.228037 16667 server_base.cc:1034] running on GCE node
I20260929 00:50:09.228742 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:09.230046 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:09.231207 16667 hybrid_clock.cc:648] HybridClock initialized: now 1790643009231176 us; error 49 us; skew 500 ppm
I20260929 00:50:09.233187 16667 webserver.cc:460] Webserver started at http://127.11.40.254:40567/ using document root <none> and password file <none>
I20260929 00:50:09.233795 16667 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:09.233858 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:09.234104 16667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:09.236220 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3cf16166a24971260a0b544d0627dc4b"
server_key_iv: "1fe28e5bb8858a70846ae7f1bdf501a6"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.236876 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3cf16166a24971260a0b544d0627dc4b"
server_key_iv: "1fe28e5bb8858a70846ae7f1bdf501a6"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.240857 16667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:50:09.243598 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:09.245173 16667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:09.245328 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3cf16166a24971260a0b544d0627dc4b"
server_key_iv: "1fe28e5bb8858a70846ae7f1bdf501a6"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.245464 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:09.256160 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:09.257031 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:09.257239 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:09.264902 16667 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33181
I20260929 00:50:09.264971 16733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33181 every 8 connection(s)
I20260929 00:50:09.266006 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:09.268735 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16667
I20260929 00:50:09.268923 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:09.269264 11427 external_mini_cluster.cc:1428] Setting key 16db4b4c88635b0c20217e672c0df661
I20260929 00:50:09.269693 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:09.277117 16734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d: Bootstrap starting.
I20260929 00:50:09.279008 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.270710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.11.40.254:33181)
0929 00:50:09.271374 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.271391 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.271430 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:50:09.272121 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.272139 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.272565 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.272953 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:09.273316 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.274302 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.275541 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.276022 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.276144 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.277076 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:09.277080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:09.277094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:09.277582 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:09.277641 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:09.277661 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:09.277743 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:09.277911 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 00:50:09.278227 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
I20260929 00:50:09.279594 16734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:09.280532 16734 log.cc:824] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:09.282244 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:50:09.282811 16734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d: No bootstrap required, opened a new log
I20260929 00:50:09.286571 16734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } }
I20260929 00:50:09.286808 16734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:09.286867 16734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5fdd70cd3d483cbc5603fdcca5d20d, State: Initialized, Role: FOLLOWER
I20260929 00:50:09.287323 16734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } }
I20260929 00:50:09.287458 16734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:09.287539 16734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:09.287637 16734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:09.288661 16734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } }
I20260929 00:50:09.289060 16734 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a5fdd70cd3d483cbc5603fdcca5d20d; no voters:
I20260929 00:50:09.289389 16734 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:09.289562 16739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:09.289839 16739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [term 1 LEADER]: Becoming Leader. State: Replica: 4a5fdd70cd3d483cbc5603fdcca5d20d, State: Running, Role: LEADER
I20260929 00:50:09.290182 16739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } }
I20260929 00:50:09.290786 16734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:09.292097 16741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5fdd70cd3d483cbc5603fdcca5d20d. Latest consensus state: current_term: 1 leader_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } } }
I20260929 00:50:09.292258 16741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.292064 16740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5fdd70cd3d483cbc5603fdcca5d20d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33181 } } }
I20260929 00:50:09.292520 16740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.292655 16748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:09.295548 16748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:09.301617 16748 catalog_manager.cc:1347] Generated new cluster ID: 75e4c2037ff54f4596ac9227a08f4633
I20260929 00:50:09.301687 16748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:09.307528 16748 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:09.308422 16748 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:09.313369 16748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a5fdd70cd3d483cbc5603fdcca5d20d: Generated new TSK 0
I20260929 00:50:09.314044 16748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:09.351910 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.328040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.11.40.254:33181)
0929 00:50:09.328197 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.328202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.328392 (+ 190us) server_negotiation.cc:407] Connection header received
0929 00:50:09.328607 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.328609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.328668 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.328771 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:50:09.329116 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.329656 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.330958 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.331193 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.331277 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.331622 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:50:09.351415 (+ 19793us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:50:09.351742 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 00:50:09.352002 16737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 00:50:09.352173 16763 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:09.327955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33181 (local address 127.0.0.1:39718)
0929 00:50:09.328289 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:09.328306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:09.328438 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:09.328801 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:09.328835 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:09.328845 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:50:09.328986 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.328994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.329798 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.329802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.330825 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.330836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.331319 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.331323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.331413 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.331444 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:50:09.351681 (+ 20237us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:50:09.351918 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.40.254:33181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
E20260929 00:50:09.352499 16764 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 00:50:09.352566 16764 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:50:09.353132 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16667
2026-09-29T00:50:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 00:50:09.392972 11427 mini_oidc.cc:152] Starting JWKS server
I20260929 00:50:09.393899 11427 webserver.cc:460] Webserver started at https://127.0.0.1:44263/ using document root <none> and password file <none>
I20260929 00:50:09.394045 11427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:50:09.394315 11427 webserver.cc:460] Webserver started at http://127.0.0.1:34035/ using document root <none> and password file <none>
2026-09-29T00:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:09Z Disabled control of system clock
I20260929 00:50:09.403908 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33817
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33817
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44263/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:50:09.519727 16770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:09.520008 16770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:09.520051 16770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:09.524119 16770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:09.524188 16770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:09.524204 16770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:09.524221 16770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:09.524236 16770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:50:09.524252 16770 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:50:09.524319 16770 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44263/jwks.json
I20260929 00:50:09.529237 16770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44263/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:09.530417 16770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:09.531905 16770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:09.540717 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:09.540714 16778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:09.540717 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:09.541054 16770 server_base.cc:1034] running on GCE node
I20260929 00:50:09.541743 16770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:09.542834 16770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:09.544000 16770 hybrid_clock.cc:648] HybridClock initialized: now 1790643009543988 us; error 28 us; skew 500 ppm
I20260929 00:50:09.545961 16770 webserver.cc:460] Webserver started at http://127.11.40.254:45015/ using document root <none> and password file <none>
I20260929 00:50:09.546671 16770 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:09.546742 16770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:09.546974 16770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:09.549189 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "8fed276ee2f34147a13952d6d0fb6a72"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "ea568ce84de57e122e6596288c246e25"
server_key_iv: "4847984791bf376c050d52c11731c3a0"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.549770 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "8fed276ee2f34147a13952d6d0fb6a72"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "ea568ce84de57e122e6596288c246e25"
server_key_iv: "4847984791bf376c050d52c11731c3a0"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.553752 16770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:50:09.556334 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:09.557718 16770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:09.557852 16770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "8fed276ee2f34147a13952d6d0fb6a72"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "ea568ce84de57e122e6596288c246e25"
server_key_iv: "4847984791bf376c050d52c11731c3a0"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.557968 16770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:09.576527 16770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:09.577385 16770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:09.577601 16770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:09.585392 16770 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33817
I20260929 00:50:09.585420 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33817 every 8 connection(s)
I20260929 00:50:09.586382 16770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:09.589685 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:09.590552 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16770
I20260929 00:50:09.590677 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:09.591010 11427 external_mini_cluster.cc:1428] Setting key c07ca6c267cf5438044fbc02a60e440f
I20260929 00:50:09.596592 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72: Bootstrap starting.
I20260929 00:50:09.599268 16837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:09.600138 16837 log.cc:824] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:09.601243 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.592589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.11.40.254:33817)
0929 00:50:09.593316 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.593332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.593371 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:50:09.594042 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.594060 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.594616 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.594998 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:09.595475 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.596473 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.597558 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.598046 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.598224 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.599150 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:09.599156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:09.599170 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:09.599657 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:09.599717 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:09.599737 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:09.599844 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:09.600007 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 00:50:09.600301 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":166,"threads_started":1}
I20260929 00:50:09.602610 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72: No bootstrap required, opened a new log
I20260929 00:50:09.604336 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:50:09.607132 16837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } }
I20260929 00:50:09.607378 16837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:09.607439 16837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fed276ee2f34147a13952d6d0fb6a72, State: Initialized, Role: FOLLOWER
I20260929 00:50:09.607858 16837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } }
I20260929 00:50:09.608002 16837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:09.608070 16837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:09.608177 16837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:09.609328 16837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } }
I20260929 00:50:09.609714 16837 leader_election.cc:302] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fed276ee2f34147a13952d6d0fb6a72; no voters:
I20260929 00:50:09.610126 16837 leader_election.cc:288] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:09.610309 16842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:09.610635 16842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [term 1 LEADER]: Becoming Leader. State: Replica: 8fed276ee2f34147a13952d6d0fb6a72, State: Running, Role: LEADER
I20260929 00:50:09.610970 16842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } }
I20260929 00:50:09.611521 16837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:09.612684 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fed276ee2f34147a13952d6d0fb6a72. Latest consensus state: current_term: 1 leader_uuid: "8fed276ee2f34147a13952d6d0fb6a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } } }
I20260929 00:50:09.612854 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.613256 16851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:09.613473 16843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fed276ee2f34147a13952d6d0fb6a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fed276ee2f34147a13952d6d0fb6a72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33817 } } }
I20260929 00:50:09.613565 16843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.616012 16851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:09.621978 16851 catalog_manager.cc:1347] Generated new cluster ID: 1c0adf7511e741788403ce7a6d8bffae
I20260929 00:50:09.622071 16851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:09.633494 16851 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:09.634486 16851 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:09.643671 16851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fed276ee2f34147a13952d6d0fb6a72: Generated new TSK 0
I20260929 00:50:09.644454 16851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:09.657730 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.652370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.11.40.254:33817)
0929 00:50:09.652524 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.652533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.652666 (+ 133us) server_negotiation.cc:407] Connection header received
0929 00:50:09.652907 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.652910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.652964 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.653066 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:50:09.653339 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.653894 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.655185 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.655430 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.655516 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.655844 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:50:09.657326 (+ 1482us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:50:09.657576 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:50:09.657806 16840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 00:50:09.659595 16866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:09.652216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33817 (local address 127.0.0.1:50440)
0929 00:50:09.652557 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:09.652576 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:09.652704 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:09.653098 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:09.653102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:09.653111 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:50:09.653226 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.653233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.654006 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.654011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.655028 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.655039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.655549 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.655553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.655643 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.655674 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:50:09.657509 (+ 1835us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:50:09.659418 (+ 1909us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.40.254:33817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1699,"thread_start_us":112,"threads_started":1}
I20260929 00:50:09.660305 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16770
E20260929 00:50:09.668743 16867 webserver.cc:533] Webserver: sslize failed:
E20260929 00:50:09.668875 16867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T00:50:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T00:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:09Z Disabled control of system clock
I20260929 00:50:09.708710 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35775
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:35775
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/testchainca.cert with env {}
W20260929 00:50:09.823278 16871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:09.823616 16871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:09.823698 16871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:09.827785 16871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:09.827895 16871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:09.827929 16871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:09.827968 16871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:09.828002 16871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:50:09.828039 16871 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:50:09.828066 16871 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 00:50:09.833702 16871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:35775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:09.834964 16871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:09.836032 16871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:09.845296 16877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:09.845331 16879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:09.845455 16871 server_base.cc:1034] running on GCE node
W20260929 00:50:09.845331 16876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:09.846251 16871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:09.847483 16871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:09.848690 16871 hybrid_clock.cc:648] HybridClock initialized: now 1790643009848662 us; error 49 us; skew 500 ppm
I20260929 00:50:09.850664 16871 webserver.cc:460] Webserver started at http://127.11.40.254:43871/ using document root <none> and password file <none>
I20260929 00:50:09.851303 16871 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:09.851404 16871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:09.851655 16871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:09.853972 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "fc3a11d28f454b0c8da15c6d6739bd7c"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3d94977f90352551c41b4e701c09690e"
server_key_iv: "b75fc6f321e0a4e35b62309f2320c77e"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.854588 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "fc3a11d28f454b0c8da15c6d6739bd7c"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3d94977f90352551c41b4e701c09690e"
server_key_iv: "b75fc6f321e0a4e35b62309f2320c77e"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.858714 16871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:09.861320 16885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:09.862618 16871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:09.862776 16871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "fc3a11d28f454b0c8da15c6d6739bd7c"
format_stamp: "Formatted at 2026-09-29 00:50:09 on dist-test-slave-5309"
server_key: "3d94977f90352551c41b4e701c09690e"
server_key_iv: "b75fc6f321e0a4e35b62309f2320c77e"
server_key_version: "encryptionkey@0"
I20260929 00:50:09.862896 16871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:09.882388 16871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:09.883246 16871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:09.883488 16871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:09.890810 16871 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:35775
I20260929 00:50:09.890815 16937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:35775 every 8 connection(s)
I20260929 00:50:09.891865 16871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:09.895020 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:09.895411 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16871
I20260929 00:50:09.895537 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:09.895843 11427 external_mini_cluster.cc:1428] Setting key 17bebd55ba1f0f7bee31645a36234324
I20260929 00:50:09.901366 16938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c: Bootstrap starting.
I20260929 00:50:09.903806 16938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:09.904738 16938 log.cc:824] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:09.906247 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.897364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.11.40.254:35775)
0929 00:50:09.898059 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.898075 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.898111 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:50:09.898757 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.898775 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.899295 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.899634 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:09.900097 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.901165 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.902151 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.902660 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.903024 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.904029 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:09.904035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:09.904055 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:09.904592 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:09.904668 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:09.904695 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:09.904851 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:09.905040 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 00:50:09.905338 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":158,"threads_started":1}
I20260929 00:50:09.907133 16938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c: No bootstrap required, opened a new log
I20260929 00:50:09.909296 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:50:09.911556 16938 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } }
I20260929 00:50:09.911806 16938 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:09.911844 16938 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3a11d28f454b0c8da15c6d6739bd7c, State: Initialized, Role: FOLLOWER
I20260929 00:50:09.912233 16938 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } }
I20260929 00:50:09.912350 16938 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:09.912411 16938 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:09.912557 16938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:09.913632 16938 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } }
I20260929 00:50:09.914003 16938 leader_election.cc:302] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3a11d28f454b0c8da15c6d6739bd7c; no voters:
I20260929 00:50:09.914422 16938 leader_election.cc:288] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:09.914510 16943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:09.914776 16943 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [term 1 LEADER]: Becoming Leader. State: Replica: fc3a11d28f454b0c8da15c6d6739bd7c, State: Running, Role: LEADER
I20260929 00:50:09.915096 16943 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } }
I20260929 00:50:09.915781 16938 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:09.916952 16944 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } } }
I20260929 00:50:09.917104 16944 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.917440 16945 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3a11d28f454b0c8da15c6d6739bd7c. Latest consensus state: current_term: 1 leader_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3a11d28f454b0c8da15c6d6739bd7c" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35775 } } }
I20260929 00:50:09.917630 16945 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:09.917500 16952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:09.920475 16952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:09.926096 16952 catalog_manager.cc:1347] Generated new cluster ID: 1bca08a58d1241eda8b38a0166001797
I20260929 00:50:09.926174 16952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:09.935322 16952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:09.936367 16952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:09.951326 16952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc3a11d28f454b0c8da15c6d6739bd7c: Generated new TSK 0
I20260929 00:50:09.952252 16952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:10.052644 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:09.968907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.11.40.254:35775)
0929 00:50:09.969063 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:09.969069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:09.969226 (+ 157us) server_negotiation.cc:407] Connection header received
0929 00:50:09.969369 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:09.969372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:09.969432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:09.969557 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:50:09.969748 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.970592 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.971455 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.971807 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.971905 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.972215 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:50:10.052127 (+ 79912us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:50:10.052486 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52780: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 00:50:10.052740 16941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52780: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 00:50:10.053169 16967 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:09.968730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35775 (local address 127.0.0.1:52780)
0929 00:50:09.969134 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:09.969153 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:09.969250 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:09.969555 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:09.969557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:09.969563 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:50:09.969653 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.969658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.970711 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.970714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.971341 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:09.971349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:09.971942 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:09.971945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:09.972012 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:09.972036 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:50:10.052431 (+ 80395us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:50:10.052637 (+ 206us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.40.254:35775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260929 00:50:10.053835 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16871
2026-09-29T00:50:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (374 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (139 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:10 dist-test-slave-5309 krb5kdc[16979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:10 dist-test-slave-5309 krb5kdc[16979](info): set up 2 sockets
Sep 29 00:50:10 dist-test-slave-5309 krb5kdc[16979](info): commencing operation
krb5kdc: starting...
W20260929 00:50:11.378641 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:11 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:50:11.508494 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35765
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:11.625218 16992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:11.625572 16992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:11.625674 16992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:11.630801 16992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:11.630916 16992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:11.630959 16992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:11.631031 16992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:11.631072 16992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:11.636665 16992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:11.637899 16992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:11.639024 16992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:11.647866 17000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:11.647931 16998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:11.647902 16997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:11.648113 16992 server_base.cc:1034] running on GCE node
I20260929 00:50:11.648793 16992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:11.650132 16992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:11.651299 16992 hybrid_clock.cc:648] HybridClock initialized: now 1790643011651265 us; error 49 us; skew 500 ppm
Sep 29 00:50:11 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643011, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:11.654213 16992 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:11.655233 16992 webserver.cc:460] Webserver started at http://127.11.40.254:39977/ using document root <none> and password file <none>
I20260929 00:50:11.655827 16992 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:11.655928 16992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:11.656178 16992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:11.658339 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "8d17ab936b164328a16b2d2e66efec3a"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "6ca141a6a4cd0192b346611f931213c8"
server_key_iv: "57d8fd1a7111b8464bdd763873595e34"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.658939 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "8d17ab936b164328a16b2d2e66efec3a"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "6ca141a6a4cd0192b346611f931213c8"
server_key_iv: "57d8fd1a7111b8464bdd763873595e34"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.663172 16992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:11.665906 17007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:11.667209 16992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:11.667364 16992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "8d17ab936b164328a16b2d2e66efec3a"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "6ca141a6a4cd0192b346611f931213c8"
server_key_iv: "57d8fd1a7111b8464bdd763873595e34"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.667520 16992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:11.688768 16992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:11.692013 16992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:11.692239 16992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:11.699628 16992 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:35765
I20260929 00:50:11.699692 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:35765 every 8 connection(s)
I20260929 00:50:11.700639 16992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:11.703538 17060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:11.705760 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 16992
I20260929 00:50:11.705926 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:11.706208 11427 external_mini_cluster.cc:1428] Setting key 468b6b8c8ee72bb8996c4b35b93839e2
I20260929 00:50:11.710006 17060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a: Bootstrap starting.
I20260929 00:50:11.712657 17060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:11.713519 17060 log.cc:824] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:50:11 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:11.716051 17060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a: No bootstrap required, opened a new log
I20260929 00:50:11.719985 17060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } }
I20260929 00:50:11.720264 17060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:11.720327 17060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d17ab936b164328a16b2d2e66efec3a, State: Initialized, Role: FOLLOWER
I20260929 00:50:11.720908 17060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } }
I20260929 00:50:11.721086 17060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:11.721172 17060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:11.721273 17060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:11.721786 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:11.707628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.11.40.254:35765)
0929 00:50:11.708049 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:11.708065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:11.708101 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:50:11.708941 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:11.708961 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:11.709317 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:11.709650 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:11.710589 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:11.711616 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:11.712872 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:11.713407 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:11.713550 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:11.716626 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:11.716631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:11.716645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:11.716673 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:11.718506 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:11.718998 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:11.719003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:11.719008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:11.719129 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:11.719442 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:11.719446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:11.719449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:11.719892 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:11.720123 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:11.720412 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 00:50:11.720764 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":137,"threads_started":1}
I20260929 00:50:11.722365 17060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } }
I20260929 00:50:11.722726 17060 leader_election.cc:302] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d17ab936b164328a16b2d2e66efec3a; no voters:
I20260929 00:50:11.723117 17060 leader_election.cc:288] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:11.723250 17065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:11.723488 17065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [term 1 LEADER]: Becoming Leader. State: Replica: 8d17ab936b164328a16b2d2e66efec3a, State: Running, Role: LEADER
I20260929 00:50:11.723836 17065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } }
I20260929 00:50:11.724465 17060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:11.725167 17066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d17ab936b164328a16b2d2e66efec3a. Latest consensus state: current_term: 1 leader_uuid: "8d17ab936b164328a16b2d2e66efec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } } }
I20260929 00:50:11.725310 17066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:11.725901 17067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d17ab936b164328a16b2d2e66efec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d17ab936b164328a16b2d2e66efec3a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35765 } } }
I20260929 00:50:11.726015 17067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:11.726265 17071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:11.729811 17071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:11.735531 17071 catalog_manager.cc:1347] Generated new cluster ID: 6ab2183841464614b3e23370007ad688
I20260929 00:50:11.735613 17071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:11.749465 17071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:11.750470 17071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:11.756350 17071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d17ab936b164328a16b2d2e66efec3a: Generated new TSK 0
I20260929 00:50:11.757144 17071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:11.800269 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:11.934499 17086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:11.934805 17086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:11.934850 17086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:11.939047 17086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:11.939118 17086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:11.939180 17086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:11.944763 17086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:11.945955 17086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:11.946990 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:11.957932 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:11.957924 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:11.957947 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:11.958282 17086 server_base.cc:1034] running on GCE node
I20260929 00:50:11.958911 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:11.959619 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:11.960881 17086 hybrid_clock.cc:648] HybridClock initialized: now 1790643011960843 us; error 48 us; skew 500 ppm
Sep 29 00:50:11 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643011, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:11.963927 17086 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:11.964884 17086 webserver.cc:460] Webserver started at http://127.11.40.193:40741/ using document root <none> and password file <none>
I20260929 00:50:11.965449 17086 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:11.965510 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:11.965680 17086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:11.967764 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "bd36a9be7b924f61baf2a12c2d0f53bb"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "c8c777bfc030ddc30cb2504de79ca82c"
server_key_iv: "5433a59ef0c2becec8b488f459adc71f"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.968302 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "bd36a9be7b924f61baf2a12c2d0f53bb"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "c8c777bfc030ddc30cb2504de79ca82c"
server_key_iv: "5433a59ef0c2becec8b488f459adc71f"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.972292 17086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:11.974840 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:11.976187 17086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:11.976332 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "bd36a9be7b924f61baf2a12c2d0f53bb"
format_stamp: "Formatted at 2026-09-29 00:50:11 on dist-test-slave-5309"
server_key: "c8c777bfc030ddc30cb2504de79ca82c"
server_key_iv: "5433a59ef0c2becec8b488f459adc71f"
server_key_version: "encryptionkey@0"
I20260929 00:50:11.976465 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:11.993491 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:11.997156 17086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:11.997417 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:11.998099 17086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:11.999001 17086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:11.999049 17086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:11.999119 17086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:11.999151 17086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.009049 17086 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45165
I20260929 00:50:12.009073 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45165 every 8 connection(s)
I20260929 00:50:12.010099 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:12.019001 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17086
I20260929 00:50:12.019169 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:12.019457 11427 external_mini_cluster.cc:1428] Setting key e2ed5d95ea1af7e926987a67cdb68206
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643011, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:12.027264 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.012224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:42483 (local address 127.11.40.254:35765)
0929 00:50:12.012396 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.012400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.013576 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 00:50:12.015280 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.015283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.015352 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.015469 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.016792 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.017390 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.018608 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.018986 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.019147 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.022682 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.022687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.022689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.022717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.024430 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.025222 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.025225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.025226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.025331 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.025840 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.025843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.025844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.026069 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.026200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.027002 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.027122 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:50:12.028661 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.012637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35765 (local address 127.11.40.193:42483)
0929 00:50:12.013231 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:12.013278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:12.015001 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:12.015671 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:12.015679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:12.016089 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:12.016614 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.016629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.017543 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.017547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.018463 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.018481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.019297 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.019301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.019697 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.020617 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:12.020637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:12.022466 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:12.024610 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.024620 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.024643 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.025094 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.025425 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.025431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.025436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.025722 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.026195 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:12.026206 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:12.026642 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 00:50:12.026940 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 00:50:12.027407 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:12.030210 17215 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35765
I20260929 00:50:12.030532 17215 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:12.031111 17215 heartbeater.cc:511] Master 127.11.40.254:35765 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:12.033176 17024 ts_manager.cc:194] Registered new tserver with Master: bd36a9be7b924f61baf2a12c2d0f53bb (127.11.40.193:45165)
I20260929 00:50:12.034415 17024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:42483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:12.052239 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:12.169914 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:12.170238 17220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:12.170288 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:12.174749 17220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:12.174832 17220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:12.174901 17220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:12.180727 17220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:12.181927 17220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:12.182981 17220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:12.193892 17226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:12.193957 17225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:12.194068 17228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:12.194417 17220 server_base.cc:1034] running on GCE node
I20260929 00:50:12.194902 17220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:12.195642 17220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:12.196880 17220 hybrid_clock.cc:648] HybridClock initialized: now 1790643012196845 us; error 51 us; skew 500 ppm
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.206126 17220 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:12.207870 17220 webserver.cc:460] Webserver started at http://127.11.40.194:44305/ using document root <none> and password file <none>
I20260929 00:50:12.208793 17220 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:12.208942 17220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:12.209252 17220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:12.212365 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "5135440887fa4e9b95caf6a6c4396602"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "d673f6eed91637c08c60b138fd846a43"
server_key_iv: "3fab001968857d3ee3adbbc4420a91b2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.213346 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "5135440887fa4e9b95caf6a6c4396602"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "d673f6eed91637c08c60b138fd846a43"
server_key_iv: "3fab001968857d3ee3adbbc4420a91b2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.219149 17220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:50:12.222764 17235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.224491 17220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:12.224648 17220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "5135440887fa4e9b95caf6a6c4396602"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "d673f6eed91637c08c60b138fd846a43"
server_key_iv: "3fab001968857d3ee3adbbc4420a91b2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.224792 17220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:12.270071 17220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:12.273970 17220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:12.274209 17220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:12.274875 17220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:12.275817 17220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:12.275893 17220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.275974 17220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:12.276029 17220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.285537 17220 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:42719
I20260929 00:50:12.285559 17348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:42719 every 8 connection(s)
I20260929 00:50:12.286511 17220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:12.292949 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17220
I20260929 00:50:12.293075 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:12.293339 11427 external_mini_cluster.cc:1428] Setting key fc59dcc4f33c1deaa64a9b12d7ae4069
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:12.301939 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.288503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:45545 (local address 127.11.40.254:35765)
0929 00:50:12.288659 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.288662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.289631 (+ 969us) server_negotiation.cc:407] Connection header received
0929 00:50:12.291143 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.291147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.291208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.291321 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.292602 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.293287 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.294806 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.295139 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.295255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.298074 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.298077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.298079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.298104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.299655 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.300211 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.300215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.300217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.300306 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.300694 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.300698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.300700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.300957 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.301065 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.301634 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.301771 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:12.302841 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.288938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35765 (local address 127.11.40.194:45545)
0929 00:50:12.289450 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:12.289506 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:12.290910 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:12.291517 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:12.291524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:12.291909 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:12.292437 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.292451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.293420 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.293430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.294640 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.294653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.295302 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.295305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.295466 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.296126 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:12.296143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:12.297872 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:12.299780 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.299787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.299805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.300096 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.300415 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.300418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.300420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.300583 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.301076 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:12.301085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:12.301354 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 00:50:12.301594 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 00:50:12.301975 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260929 00:50:12.304181 17349 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35765
I20260929 00:50:12.304464 17349 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:12.305277 17349 heartbeater.cc:511] Master 127.11.40.254:35765 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:50:12.306555 17024 ts_manager.cc:194] Registered new tserver with Master: 5135440887fa4e9b95caf6a6c4396602 (127.11.40.194:42719)
I20260929 00:50:12.307157 17024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:45545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:12.324258 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:12.439764 17354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:12.440079 17354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:12.440124 17354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:12.444475 17354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:12.444553 17354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:12.444635 17354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:12.450248 17354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:12.451493 17354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:12.452550 17354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:12.462880 17362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:12.462880 17359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:12.462891 17360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:12.463284 17354 server_base.cc:1034] running on GCE node
I20260929 00:50:12.463757 17354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:12.464458 17354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:12.465674 17354 hybrid_clock.cc:648] HybridClock initialized: now 1790643012465630 us; error 62 us; skew 500 ppm
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.468750 17354 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:12.469708 17354 webserver.cc:460] Webserver started at http://127.11.40.195:46855/ using document root <none> and password file <none>
I20260929 00:50:12.470371 17354 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:12.470467 17354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:12.470728 17354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:12.473090 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "4e739a77be944a4d850d24e5c9ce5819"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "aeab76fba60d56a9ffe9e054110fdcf4"
server_key_iv: "ac1fb1ab8fd2ce788ee9ed207e347eb2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.473692 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "4e739a77be944a4d850d24e5c9ce5819"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "aeab76fba60d56a9ffe9e054110fdcf4"
server_key_iv: "ac1fb1ab8fd2ce788ee9ed207e347eb2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.477639 17354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:12.480239 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.481758 17354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:12.481909 17354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "4e739a77be944a4d850d24e5c9ce5819"
format_stamp: "Formatted at 2026-09-29 00:50:12 on dist-test-slave-5309"
server_key: "aeab76fba60d56a9ffe9e054110fdcf4"
server_key_iv: "ac1fb1ab8fd2ce788ee9ed207e347eb2"
server_key_version: "encryptionkey@0"
I20260929 00:50:12.482041 17354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:12.503263 17354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:12.506485 17354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:12.506707 17354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:12.507361 17354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:12.508232 17354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:12.508306 17354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.508373 17354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:12.508422 17354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:12.518123 17354 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:33339
I20260929 00:50:12.518150 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:33339 every 8 connection(s)
I20260929 00:50:12.519088 17354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:12.521836 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17354
I20260929 00:50:12.521955 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:12.522261 11427 external_mini_cluster.cc:1428] Setting key 84815cd18c277c83d5c3ca7e3b25f6de
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:12.534629 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.521048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:34331 (local address 127.11.40.254:35765)
0929 00:50:12.521198 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.521201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.522301 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 00:50:12.524013 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.524017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.524076 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.524159 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.525582 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.526237 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.527725 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.527964 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.528087 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.530985 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.530989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.530991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.531018 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.532419 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.533042 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.533046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.533047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.533127 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.533488 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.533491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.533493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.533695 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.533814 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.534328 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.534463 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:12.535632 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.521546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35765 (local address 127.11.40.195:34331)
0929 00:50:12.522135 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:12.522182 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:12.523792 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:12.524349 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:12.524356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:12.524918 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:12.525425 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.525439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.526369 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.526372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.527572 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:12.527584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.528132 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.528136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:12.528335 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.529080 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:12.529096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:12.530784 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:12.532551 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.532559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.532576 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.532920 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.533224 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:12.533227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:12.533229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:12.533380 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:12.533812 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:12.533819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:12.534083 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 00:50:12.534275 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 00:50:12.534630 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":162,"threads_started":1}
I20260929 00:50:12.536931 17483 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35765
I20260929 00:50:12.537204 17483 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:12.537752 17483 heartbeater.cc:511] Master 127.11.40.254:35765 requested a full tablet report, sending...
I20260929 00:50:12.538868 17023 ts_manager.cc:194] Registered new tserver with Master: 4e739a77be944a4d850d24e5c9ce5819 (127.11.40.195:33339)
I20260929 00:50:12.539506 17023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:34331
I20260929 00:50:12.548893 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/test-admin.keytab.
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.566711 11427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 00:50:12.569859 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:12.568269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.11.40.193:45165)
0929 00:50:12.568658 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.568665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.568687 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:50:12.568737 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.568742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.569014 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.569178 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.569697 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40692: BlockingRecv error: recv got EOF from 127.0.0.1:40692 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
W20260929 00:50:12.570065 11427 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.571255 11427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.572326 11427 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:50:12.582712 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.573332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.11.40.193:45165)
0929 00:50:12.573477 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.573480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.573495 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:50:12.573617 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.573619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.573662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.573733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.574272 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.575063 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.575899 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.576288 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.576376 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.578855 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.578866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.578881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.578928 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.580994 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.581422 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.581427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.581429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.581510 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.581745 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.581747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.581748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.582010 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.582169 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.582352 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.582482 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 00:50:12.587005 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:12.585909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.11.40.193:45165)
0929 00:50:12.586042 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.586046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.586129 (+ 83us) server_negotiation.cc:407] Connection header received
0929 00:50:12.586247 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.586250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.586290 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.586360 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.586850 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40710: BlockingRecv error: recv got EOF from 127.0.0.1:40710 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 00:50:12.587055 11427 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.588186 11427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.589313 11427 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:50:12.601691 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.590524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.11.40.193:45165)
0929 00:50:12.590710 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.590715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.590947 (+ 232us) server_negotiation.cc:407] Connection header received
0929 00:50:12.591127 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.591131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.591180 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.591294 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.592039 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.592986 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.594326 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.594737 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.594870 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.598463 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.598467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.598469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.598501 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.599870 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.600325 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.600329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.600330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.600418 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.600776 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.600779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.600780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.601084 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.601221 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.601441 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.601563 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:50:12.604759 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:12.603660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.11.40.193:45165)
0929 00:50:12.603813 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.603817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.603927 (+ 110us) server_negotiation.cc:407] Connection header received
0929 00:50:12.604081 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.604084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.604134 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.604225 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.604636 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40722: BlockingRecv error: recv got EOF from 127.0.0.1:40722 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 00:50:12.604949 11427 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790642980693194-11427-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.606112 11427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:12.607046 11427 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[16979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:50:12.615847 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:12.607939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.11.40.193:45165)
0929 00:50:12.608087 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:12.608090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:12.608139 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:50:12.608250 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:12.608252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:12.608288 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:12.608365 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:12.608919 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.609435 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.610203 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:12.610424 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:12.610508 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:12.613159 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:12.613162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:12.613164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:12.613188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:12.614431 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.614819 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.614822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.614824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.614892 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:12.615128 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:12.615130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:12.615131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:12.615286 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:12.615405 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:12.615558 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 00:50:12.615648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:50:12.617007 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17086
I20260929 00:50:12.623903 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17220
I20260929 00:50:12.629694 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17354
I20260929 00:50:12.635358 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 16992
2026-09-29T00:50:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2314 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[17503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[17503](info): set up 2 sockets
Sep 29 00:50:12 dist-test-slave-5309 krb5kdc[17503](info): commencing operation
krb5kdc: starting...
W20260929 00:50:13.686805 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:13 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab.
Entry for principal oryx/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab.
I20260929 00:50:13.816378 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:40939
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab
--principal=oryx/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:13.932436 17516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:13.932802 17516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:13.932909 17516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:13.937386 17516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:13.937532 17516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:13.937577 17516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:13.937619 17516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:13.937651 17516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:13.943363 17516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:40939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab
--principal=oryx/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:13.944638 17516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:13.945814 17516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:50:13.955250 17516 server_base.cc:1034] running on GCE node
W20260929 00:50:13.955276 17522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:13.955276 17521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:13.955502 17524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:13.956177 17516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:13.957533 17516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:13.958737 17516 hybrid_clock.cc:648] HybridClock initialized: now 1790643013958713 us; error 48 us; skew 500 ppm
Sep 29 00:50:13 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:13.961791 17516 init.cc:377] Logged in from keytab as oryx/127.11.40.254@KRBTEST.COM (short username oryx)
I20260929 00:50:13.962865 17516 webserver.cc:460] Webserver started at http://127.11.40.254:43047/ using document root <none> and password file <none>
I20260929 00:50:13.963544 17516 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:13.963644 17516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:13.963920 17516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:13.966192 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "f08ab8798c944c868d040a4f58fdee83"
format_stamp: "Formatted at 2026-09-29 00:50:13 on dist-test-slave-5309"
server_key: "c0347dc30fa055564367573065618437"
server_key_iv: "f24ac124acb6bd2124c738b2c47567ae"
server_key_version: "encryptionkey@0"
I20260929 00:50:13.966884 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "f08ab8798c944c868d040a4f58fdee83"
format_stamp: "Formatted at 2026-09-29 00:50:13 on dist-test-slave-5309"
server_key: "c0347dc30fa055564367573065618437"
server_key_iv: "f24ac124acb6bd2124c738b2c47567ae"
server_key_version: "encryptionkey@0"
I20260929 00:50:13.971184 17516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:13.973992 17531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:13.975519 17516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:13.975682 17516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "f08ab8798c944c868d040a4f58fdee83"
format_stamp: "Formatted at 2026-09-29 00:50:13 on dist-test-slave-5309"
server_key: "c0347dc30fa055564367573065618437"
server_key_iv: "f24ac124acb6bd2124c738b2c47567ae"
server_key_version: "encryptionkey@0"
I20260929 00:50:13.975831 17516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:13.991900 17516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:13.995633 17516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:13.995868 17516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:14.004464 17516 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:40939
I20260929 00:50:14.004878 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:40939 every 8 connection(s)
I20260929 00:50:14.005661 17516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:14.008668 17584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:14.013801 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17516
I20260929 00:50:14.014004 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:14.014289 11427 external_mini_cluster.cc:1428] Setting key ea1e57e9258a7f7c694d7d1a4f4bae1d
I20260929 00:50:14.014973 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Bootstrap starting.
I20260929 00:50:14.017537 17584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:14.018378 17584 log.cc:824] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:14.020437 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: No bootstrap required, opened a new log
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:14.024663 17584 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } }
I20260929 00:50:14.024984 17584 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:14.025079 17584 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08ab8798c944c868d040a4f58fdee83, State: Initialized, Role: FOLLOWER
I20260929 00:50:14.025588 17584 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } }
I20260929 00:50:14.025786 17584 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:14.025866 17584 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:14.026054 17584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:14.027149 17584 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } }
I20260929 00:50:14.027595 17584 leader_election.cc:302] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08ab8798c944c868d040a4f58fdee83; no voters:
I20260929 00:50:14.028000 17584 leader_election.cc:288] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:14.028126 17589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:14.028398 17589 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 LEADER]: Becoming Leader. State: Replica: f08ab8798c944c868d040a4f58fdee83, State: Running, Role: LEADER
I20260929 00:50:14.028784 17589 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } }
I20260929 00:50:14.029507 17584 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:14.030718 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } }
I20260929 00:50:14.030857 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:14.030942 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.015764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.11.40.254:40939)
0929 00:50:14.016261 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:14.016282 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:14.016516 (+ 234us) server_negotiation.cc:407] Connection header received
0929 00:50:14.017400 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:14.017419 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:14.017858 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:14.018253 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:14.019073 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.020080 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.021068 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.021556 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.021736 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.024601 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:14.024607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:14.024621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:14.024651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:14.026727 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.027364 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.027371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.027378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.027517 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.027866 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.027870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.027874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.028447 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:14.028645 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:14.029484 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 00:50:14.029812 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":176,"threads_started":1}
I20260929 00:50:14.031252 17598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:14.030722 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ab8798c944c868d040a4f58fdee83. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } }
I20260929 00:50:14.031417 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:14.034705 17598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:14.041229 17598 catalog_manager.cc:1347] Generated new cluster ID: 71fae3a7ca6b4a8eb90f5b99d5adcccb
I20260929 00:50:14.041329 17598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:14.067188 17598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:14.068661 17598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:14.075183 17598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Generated new TSK 0
I20260929 00:50:14.075924 17598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
I20260929 00:50:14.110049 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:14.234328 17610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:14.234640 17610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:14.234687 17610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:14.239531 17610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:14.239635 17610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:14.239743 17610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:14.246203 17610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:14.247463 17610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:14.248605 17610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:14.259287 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:14.259342 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:14.259308 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:14.259531 17610 server_base.cc:1034] running on GCE node
I20260929 00:50:14.260020 17610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:14.260718 17610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:14.261942 17610 hybrid_clock.cc:648] HybridClock initialized: now 1790643014261924 us; error 31 us; skew 500 ppm
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:14.265228 17610 init.cc:377] Logged in from keytab as oryx/127.11.40.193@KRBTEST.COM (short username oryx)
I20260929 00:50:14.266129 17610 webserver.cc:460] Webserver started at http://127.11.40.193:34529/ using document root <none> and password file <none>
I20260929 00:50:14.266680 17610 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:14.266741 17610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:14.266983 17610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:14.269281 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "ad357811cbfb0e2db96b4b18e8fa6ac1"
server_key_iv: "30aa376eb135be05ca51b458e4026500"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.269814 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "ad357811cbfb0e2db96b4b18e8fa6ac1"
server_key_iv: "30aa376eb135be05ca51b458e4026500"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.273790 17610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:14.276307 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.277745 17610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:14.277866 17610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "ad357811cbfb0e2db96b4b18e8fa6ac1"
server_key_iv: "30aa376eb135be05ca51b458e4026500"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.277964 17610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:14.298573 17610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:14.302158 17610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:14.302380 17610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:14.303013 17610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:14.303884 17610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:14.303958 17610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.304026 17610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:14.304069 17610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.313330 17610 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:38333
I20260929 00:50:14.313356 17738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:38333 every 8 connection(s)
I20260929 00:50:14.314289 17610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:14.318179 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17610
I20260929 00:50:14.318326 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:14.318696 11427 external_mini_cluster.cc:1428] Setting key 871f523be1d1240793416132c2d040eb
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:14.331525 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.316433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:34659 (local address 127.11.40.254:40939)
0929 00:50:14.316659 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:14.316662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:14.317632 (+ 970us) server_negotiation.cc:407] Connection header received
0929 00:50:14.319414 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:14.319417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:14.319476 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:14.319556 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:14.321069 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.321682 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.322978 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.323228 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.323357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.326562 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:14.326567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:14.326572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:14.326609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:14.328561 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.329452 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.329459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.329466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.329559 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.330070 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.330076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.330078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.330435 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:14.330550 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:14.331129 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 00:50:14.331373 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:14.332617 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.316911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:34659)
0929 00:50:14.317467 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:14.317514 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:14.319185 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:14.319728 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:14.319735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:14.320108 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:14.320881 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.320907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.321873 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.321876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.322836 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.322846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.323407 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.323412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.323601 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.324368 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:14.324385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:14.326335 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:14.328742 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.328749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.328765 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.329307 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.329656 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.329663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.329664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.329835 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.330564 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:14.330570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:14.330849 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 00:50:14.331042 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 00:50:14.331437 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260929 00:50:14.334134 17739 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:14.334416 17739 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:14.334951 17739 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
I20260929 00:50:14.336651 17547 ts_manager.cc:194] Registered new tserver with Master: e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193:38333)
I20260929 00:50:14.337926 17547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:34659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
I20260929 00:50:14.348743 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:14.462248 17744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:14.462649 17744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:14.462735 17744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:14.467002 17744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:14.467121 17744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:14.467221 17744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:14.473078 17744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:14.474372 17744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:14.475553 17744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:14.485740 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:14.485749 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:14.485749 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:14.486114 17744 server_base.cc:1034] running on GCE node
I20260929 00:50:14.486544 17744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:14.487236 17744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:14.488425 17744 hybrid_clock.cc:648] HybridClock initialized: now 1790643014488395 us; error 44 us; skew 500 ppm
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:14.491711 17744 init.cc:377] Logged in from keytab as oryx/127.11.40.194@KRBTEST.COM (short username oryx)
I20260929 00:50:14.492844 17744 webserver.cc:460] Webserver started at http://127.11.40.194:36289/ using document root <none> and password file <none>
I20260929 00:50:14.493471 17744 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:14.493541 17744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:14.493788 17744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:14.496017 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "312dcaf687e64010a9db6a66b73f068f"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "e204d23e7c943b892d0f8250d69b517d"
server_key_iv: "f96a20738d4dbd255db9e7a5bf39e4b6"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.496637 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "312dcaf687e64010a9db6a66b73f068f"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "e204d23e7c943b892d0f8250d69b517d"
server_key_iv: "f96a20738d4dbd255db9e7a5bf39e4b6"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.501201 17744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 00:50:14.503902 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.505415 17744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:14.505522 17744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "312dcaf687e64010a9db6a66b73f068f"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "e204d23e7c943b892d0f8250d69b517d"
server_key_iv: "f96a20738d4dbd255db9e7a5bf39e4b6"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.505659 17744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:14.559105 17744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:14.563602 17744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:14.563872 17744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:14.564551 17744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:14.565609 17744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:14.565658 17744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.565701 17744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:14.565716 17744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.576151 17744 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:35963
I20260929 00:50:14.576160 17872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:35963 every 8 connection(s)
I20260929 00:50:14.577422 17744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:14.586941 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17744
I20260929 00:50:14.587090 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:14.587473 11427 external_mini_cluster.cc:1428] Setting key c82ef81456be11a30725a87afcb17b57
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:14.593365 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.579462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:57257 (local address 127.11.40.254:40939)
0929 00:50:14.579605 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:14.579609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:14.580554 (+ 945us) server_negotiation.cc:407] Connection header received
0929 00:50:14.582240 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:14.582243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:14.582320 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:14.582398 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:14.583748 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.584300 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.585577 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.585824 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.585943 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.589405 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:14.589413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:14.589415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:14.589442 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:14.591036 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.591632 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.591641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.591642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.591723 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.592116 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.592119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.592121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.592325 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:14.592442 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:14.593058 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 00:50:14.593198 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:50:14.594292 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.579862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:57257)
0929 00:50:14.580388 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:14.580437 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:14.582010 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:14.582616 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:14.582627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:14.583022 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:14.583589 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.583603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.584443 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.584446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.585444 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.585453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.585992 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.585996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.586172 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.586826 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:14.586850 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:14.589139 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:14.591172 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.591180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.591200 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.591504 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.591836 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.591840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.591842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.592004 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.592442 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:14.592450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:14.592736 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 00:50:14.592997 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 00:50:14.593379 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":162,"threads_started":1}
I20260929 00:50:14.595638 17873 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:14.595947 17873 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:14.596634 17873 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:14.598061 17547 ts_manager.cc:194] Registered new tserver with Master: 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
I20260929 00:50:14.599089 17547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:57257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
I20260929 00:50:14.621949 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:14.739264 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:14.739617 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:14.739701 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:14.743916 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:14.744042 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:14.744146 17878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:14.749954 17878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:14.751173 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:14.752368 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:14.762914 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:14.762914 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:14.763309 17878 server_base.cc:1034] running on GCE node
W20260929 00:50:14.763414 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:14.763918 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:14.764664 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:14.765836 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790643014765807 us; error 39 us; skew 500 ppm
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:14.768966 17878 init.cc:377] Logged in from keytab as oryx/127.11.40.195@KRBTEST.COM (short username oryx)
I20260929 00:50:14.769989 17878 webserver.cc:460] Webserver started at http://127.11.40.195:41251/ using document root <none> and password file <none>
I20260929 00:50:14.770591 17878 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:14.770686 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:14.770998 17878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:14.773262 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "47a69599c9e242d19b026d29b96d3e7b"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "568826de0c4aa4b7dfbe8d2bd00cde61"
server_key_iv: "518c76caa3b69c6dd5d0f4e06ab5c95a"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.773877 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "47a69599c9e242d19b026d29b96d3e7b"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "568826de0c4aa4b7dfbe8d2bd00cde61"
server_key_iv: "518c76caa3b69c6dd5d0f4e06ab5c95a"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.778016 17878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:14.780791 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.782545 17878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:14.782671 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "47a69599c9e242d19b026d29b96d3e7b"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "568826de0c4aa4b7dfbe8d2bd00cde61"
server_key_iv: "518c76caa3b69c6dd5d0f4e06ab5c95a"
server_key_version: "encryptionkey@0"
I20260929 00:50:14.782843 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:14.795272 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:14.799136 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:14.799374 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:14.800032 17878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:14.801028 17878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:14.801100 17878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.801177 17878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:14.801230 17878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:14.811120 17878 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:43241
I20260929 00:50:14.811142 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:43241 every 8 connection(s)
I20260929 00:50:14.812199 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:14.819811 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 17878
I20260929 00:50:14.819972 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:14.820327 11427 external_mini_cluster.cc:1428] Setting key 7ca20cf426608e9df594a701fa26f44b
Sep 29 00:50:14 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643014, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:14.827539 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.814181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:59419 (local address 127.11.40.254:40939)
0929 00:50:14.814333 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:14.814337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:14.815294 (+ 957us) server_negotiation.cc:407] Connection header received
0929 00:50:14.816775 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:14.816779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:14.816904 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:14.817004 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:14.818255 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.818801 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.820386 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.820836 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.820964 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.823848 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:14.823851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:14.823853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:14.823890 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:14.825525 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.826086 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.826089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.826090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.826181 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:14.826517 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:14.826519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:14.826520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:14.826706 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:14.826777 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:14.827307 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 00:50:14.827414 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:50:14.828590 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:14.814566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:59419)
0929 00:50:14.815113 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:14.815179 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:14.816542 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:14.817203 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:14.817210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:14.817596 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:14.818101 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.818115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.818948 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.818951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.820177 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:14.820189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:14.821065 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:14.821070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:14.821309 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:14.822057 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:14.822076 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:14.823644 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:14.825650 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.825660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.825674 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.825987 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.826272 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:14.826275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:14.826276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:14.826423 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:14.826801 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:14.826807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:14.827096 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 00:50:14.827262 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 00:50:14.827545 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260929 00:50:14.829921 18007 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:14.830219 18007 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:14.830849 18007 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
I20260929 00:50:14.832055 17547 ts_manager.cc:194] Registered new tserver with Master: 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:14.832612 17547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:59419
I20260929 00:50:14.834690 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.253.keytab.
Entry for principal oryx/127.11.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.253.keytab.
I20260929 00:50:14.865819 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:39543
--webserver_interface=127.11.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.253.keytab
--principal=oryx/127.11.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:40939,127.11.40.253:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:14.982142 18012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:14.982439 18012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:14.982542 18012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:14.986750 18012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:14.986855 18012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:14.986891 18012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:14.986943 18012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:14.987000 18012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:14.992652 18012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:40939,127.11.40.253:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.253.keytab
--principal=oryx/127.11.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
W20260929 00:50:14.993014 18012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.40.254:40939,127.11.40.253:39543'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 00:50:14.994001 18012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:14.995090 18012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:15.005360 18017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.005366 18020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.005362 18018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.009169 18012 server_base.cc:1034] running on GCE node
I20260929 00:50:15.010030 18012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:15.011420 18012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:15.012666 18012 hybrid_clock.cc:648] HybridClock initialized: now 1790643015012647 us; error 48 us; skew 500 ppm
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:15.016072 18012 init.cc:377] Logged in from keytab as oryx/127.11.40.253@KRBTEST.COM (short username oryx)
I20260929 00:50:15.017313 18012 webserver.cc:460] Webserver started at http://127.11.40.253:45423/ using document root <none> and password file <none>
I20260929 00:50:15.017933 18012 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.018030 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.018280 18012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:15.020498 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/instance:
uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "4ed269854f7dfc709b42b5745f230f6f"
server_key_iv: "86bab36619f238d120237a60594789c8"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.021152 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal/instance:
uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "4ed269854f7dfc709b42b5745f230f6f"
server_key_iv: "86bab36619f238d120237a60594789c8"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.025461 18012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:15.028277 18027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.029865 18012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:15.030030 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "4ed269854f7dfc709b42b5745f230f6f"
server_key_iv: "86bab36619f238d120237a60594789c8"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.030217 18012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.059177 18012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:15.062578 18012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:15.062803 18012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:15.070854 18012 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.253:39543
I20260929 00:50:15.071154 18079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.253:39543 every 8 connection(s)
I20260929 00:50:15.071919 18012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
I20260929 00:50:15.073375 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18012
I20260929 00:50:15.075116 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:15.079960 18080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.253@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:15.089632 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.074983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.11.40.253:39543)
0929 00:50:15.075445 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.075467 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.075509 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:50:15.076347 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.076370 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.076806 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.077201 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.077903 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.079428 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.080386 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.081097 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.081277 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.084385 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.084390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.084399 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.084428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.086442 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.086962 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.086995 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.086999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.087115 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.087475 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.087481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.087486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.087946 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.088165 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.088475 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.088765 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260929 00:50:15.091611 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.081856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.11.40.254:40939)
0929 00:50:15.081978 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.081981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.082350 (+ 369us) server_negotiation.cc:407] Connection header received
0929 00:50:15.082479 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.082482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.082551 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.082658 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.083416 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.084010 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.085211 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.085571 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.085717 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.088220 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.088226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.088233 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.088261 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.089876 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.090349 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.090352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.090354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.090440 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.090736 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.090738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.090740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.090958 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.091051 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.091387 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.091494 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 00:50:15.091739 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.081801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49802)
0929 00:50:15.082201 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.082251 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.082374 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.082723 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.082732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.083106 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.083270 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.083280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.084144 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.084150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.085094 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.085103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.085819 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.085824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.085939 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.086450 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.086462 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.088073 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.089991 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.089999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.090004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.090232 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.090535 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.090540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.090542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.090634 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.091075 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.091081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.091194 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.091335 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.091506 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":152,"threads_started":1}
I20260929 00:50:15.093045 18080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.253@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
I20260929 00:50:15.101074 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.093584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50198)
0929 00:50:15.093695 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.093707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.093770 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.093967 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.093969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.094099 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.094187 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.094192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.095242 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.095244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.095760 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.095768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.096155 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.096157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.096232 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.096733 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.096744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.098068 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.099626 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.099628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.099629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.099915 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.100185 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.100187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.100188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.100259 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.100590 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.100592 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.100666 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.100749 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.100925 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 00:50:15.101107 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.093630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.11.40.253:39543)
0929 00:50:15.093745 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.093748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.093759 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:50:15.093840 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.093842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.093877 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.093940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.094274 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.095140 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.095855 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.096065 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.096138 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.098198 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.098201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.098204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.098224 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.099505 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.100021 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.100024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.100025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.100101 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.100335 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.100337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.100338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.100511 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.100585 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.100802 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.100987 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:50:15.104187 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Bootstrap starting.
I20260929 00:50:15.106590 18080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:15.107357 18080 log.cc:824] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:15.109614 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: No bootstrap required, opened a new log
I20260929 00:50:15.112708 18080 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } }
I20260929 00:50:15.113031 18080 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:15.113127 18080 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ef0b11ae2d4ed0b3a40220fd2c32a3, State: Initialized, Role: FOLLOWER
I20260929 00:50:15.113655 18080 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } }
I20260929 00:50:15.114197 18086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } }
I20260929 00:50:15.114326 18086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:15.114925 18080 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:50:15.120008 18097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:50:15.120107 18097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:50:15.122442 18012 master_runner.cc:428] Detected that this master f7ef0b11ae2d4ed0b3a40220fd2c32a3 is joining an existing cluster
I20260929 00:50:15.122514 18012 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.40.253:39543
I20260929 00:50:15.133709 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.124775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50210)
0929 00:50:15.125618 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.125630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.125737 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.125979 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.125983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.126213 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.126344 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.126351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.127297 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.127299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.127876 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.127884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.128454 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.128553 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.129131 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.129151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.129798 (+ 647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.131985 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.131990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.131994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.132252 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.132522 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.132527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.132616 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.133060 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.133063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.133172 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.133280 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.133488 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":108,"threads_started":1}
I20260929 00:50:15.133998 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.125104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.11.40.254:40939)
0929 00:50:15.125240 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.125244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.125390 (+ 146us) server_negotiation.cc:407] Connection header received
0929 00:50:15.125523 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.125525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.125565 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.125663 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.126207 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.126994 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128194 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.128533 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128645 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.129926 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.129930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.129933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.129959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.132166 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132592 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.132596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.132597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.132703 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.133029 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.133035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.133036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.133223 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.133375 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.133753 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.133876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:50:15.133816 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.124712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.11.40.253:39543)
0929 00:50:15.124885 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.124892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.125688 (+ 796us) server_negotiation.cc:407] Connection header received
0929 00:50:15.125812 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.125816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.125858 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.125942 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.126475 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.127187 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128003 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.128320 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128419 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.129933 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.129936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.129939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.129959 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.131881 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132345 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.132351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.132353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.132426 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132708 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.132711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.132713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.132969 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.133061 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.133351 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.133578 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":50}
I20260929 00:50:15.134382 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.124690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49816)
0929 00:50:15.125299 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.125316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.125431 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.125696 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.125699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.125951 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.126082 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.126095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.127111 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.127115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.128074 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.128085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.128647 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.128651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.128742 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.129344 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.129358 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.129772 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.132268 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.132275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.132518 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.132811 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.132839 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.132841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.132916 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.133446 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.133448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.133568 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.134117 (+ 549us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.134220 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1,"thread_start_us":155,"threads_started":1}
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.254@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
I20260929 00:50:15.149384 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.139284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.11.40.253:39543)
0929 00:50:15.139427 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.139431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.139828 (+ 397us) server_negotiation.cc:407] Connection header received
0929 00:50:15.140005 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.140007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.140050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.140116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.140910 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.141395 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.142650 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.142939 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.143041 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.145870 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.145873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.145876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.145908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.147386 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.147933 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.147944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.147945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.148020 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.148312 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.148317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.148319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.148537 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.148649 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.149095 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.149218 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:50:15.149423 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.139251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50214)
0929 00:50:15.139685 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.139743 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.139878 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.140172 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.140179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.140640 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.140775 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.140783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.141498 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.141502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.142512 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.142522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.143058 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.143062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.143188 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.143779 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.143794 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.145712 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.147514 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.147524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.147530 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.147807 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.148123 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.148126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.148128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.148219 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.148657 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.148667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.148795 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.149031 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.149241 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":124,"threads_started":1}
I20260929 00:50:15.150943 17547 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.40.253:39543: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
member_type: NON_VOTER
last_known_addr {
host: "127.11.40.253"
port: 39543
}
attrs {
promote: true
}
}
dest_uuid: "f08ab8798c944c868d040a4f58fdee83"
cas_config_opid_index: -1
I20260929 00:50:15.151602 17547 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } }
I20260929 00:50:15.153575 18108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.154570 18054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.11.40.252@KRBTEST.COM; defaulting to no policy
I20260929 00:50:15.155666 18108 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.40.253:39543
I20260929 00:50:15.156080 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.156191 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:15.156216 18012 master.cc:537] Master@127.11.40.253:39543 shutting down...
I20260929 00:50:15.156188 18054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 00:50:15.157539 18109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.252@KRBTEST.COM" created.
W20260929 00:50:15.161868 17534 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 00:50:15.162391 18012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:15.162783 18012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:15.162937 18012 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: stopping tablet replica
W20260929 00:50:15.163002 17534 consensus_peers.cc:487] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 -> Peer f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253:39543): Couldn't send request to peer f7ef0b11ae2d4ed0b3a40220fd2c32a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:50:15.168170 18012 master.cc:552] Master@127.11.40.253:39543 shutdown complete.
I20260929 00:50:15.168229 18012 master_runner.cc:305] Clearing existing system tablet
I20260929 00:50:15.168497 18012 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.168576 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.169981 18012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:15.171090 18114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.171340 18012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.171496 18012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:15.171541 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "4ed269854f7dfc709b42b5745f230f6f"
server_key_iv: "86bab36619f238d120237a60594789c8"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.173765 18012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:15.176775 18012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:50:15.176927 18012 log.cc:1196] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 00:50:15.177570 18012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Deleting consensus metadata
I20260929 00:50:15.177773 18012 master_runner.cc:315] Copying system tablet from 127.11.40.254:40939
I20260929 00:50:15.179121 18012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.40.254:40939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.252.keytab.
Entry for principal oryx/127.11.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.252.keytab.
I20260929 00:50:15.181430 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:34399
--webserver_interface=127.11.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.252.keytab
--principal=oryx/127.11.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:50:15.189033 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.179902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.11.40.254:40939)
0929 00:50:15.180039 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.180043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.180130 (+ 87us) server_negotiation.cc:407] Connection header received
0929 00:50:15.180325 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.180328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.180382 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.180479 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.181183 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.182046 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.183237 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.183586 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.183662 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.185544 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.185548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.185595 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.185622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.187159 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.187678 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.187692 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.187694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.187788 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.188069 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.188073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.188075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.188268 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.188381 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.188654 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.188776 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:15.189079 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.179763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49828)
0929 00:50:15.180020 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.180035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.180211 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.180604 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.180607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.180913 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.181058 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.181066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.182169 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.182173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.183091 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.183102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.183829 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.183832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.183940 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.185034 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.185047 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.185383 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.187308 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.187313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.187315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.187537 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.187887 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.187889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.187891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.187973 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.188379 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.188382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.188484 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.188594 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.188760 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
I20260929 00:50:15.189950 17568 tablet_copy_service.cc:140] P f08ab8798c944c868d040a4f58fdee83: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7ef0b11ae2d4ed0b3a40220fd2c32a3 ({username='oryx', principal='oryx/127.11.40.253@KRBTEST.COM'} at 127.0.0.1:49828)
I20260929 00:50:15.190145 17568 tablet_copy_service.cc:161] P f08ab8798c944c868d040a4f58fdee83: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7ef0b11ae2d4ed0b3a40220fd2c32a3 at {username='oryx', principal='oryx/127.11.40.253@KRBTEST.COM'} at 127.0.0.1:49828: session id = f7ef0b11ae2d4ed0b3a40220fd2c32a3-00000000000000000000000000000000
I20260929 00:50:15.191572 17568 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:50:15.193181 18012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:50:15.195104 18012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: tablet copy: Starting download of 0 data blocks...
I20260929 00:50:15.195318 18012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: tablet copy: Starting download of 1 WAL segments...
I20260929 00:50:15.197042 18012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:50:15.198683 17568 tablet_copy_service.cc:342] P f08ab8798c944c868d040a4f58fdee83: Request end of tablet copy session f7ef0b11ae2d4ed0b3a40220fd2c32a3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.40.253@KRBTEST.COM'} at 127.0.0.1:49828
I20260929 00:50:15.198805 17568 tablet_copy_service.cc:433] P f08ab8798c944c868d040a4f58fdee83: ending tablet copy session f7ef0b11ae2d4ed0b3a40220fd2c32a3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7ef0b11ae2d4ed0b3a40220fd2c32a3
I20260929 00:50:15.205926 18012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:15.207813 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.207818 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.208328 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.208997 18012 server_base.cc:1034] running on GCE node
I20260929 00:50:15.209204 18012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:15.209544 18012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:15.210683 18012 hybrid_clock.cc:648] HybridClock initialized: now 1790643015210683 us; error 50 us; skew 500 ppm
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:15.212218 18012 init.cc:377] Logged in from keytab as oryx/127.11.40.253@KRBTEST.COM (short username oryx)
I20260929 00:50:15.212759 18012 webserver.cc:460] Webserver started at http://127.11.40.253:37947/ using document root <none> and password file <none>
I20260929 00:50:15.213008 18012 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.213068 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.214221 18012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:50:15.215199 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.215452 18012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:50:15.215550 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "4ed269854f7dfc709b42b5745f230f6f"
server_key_iv: "86bab36619f238d120237a60594789c8"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.215659 18012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.226125 18012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:15.228370 18012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:15.233582 18012 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.253:39543
I20260929 00:50:15.234259 18012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
I20260929 00:50:15.235786 18185 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:50:15.236761 18185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Bootstrap starting.
I20260929 00:50:15.239836 18184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.253:39543 every 8 connection(s)
I20260929 00:50:15.246400 18185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:50:15.246788 18185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: Bootstrap complete.
I20260929 00:50:15.247330 18185 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } }
I20260929 00:50:15.247474 18185 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7ef0b11ae2d4ed0b3a40220fd2c32a3, State: Initialized, Role: LEARNER
I20260929 00:50:15.247591 18185 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } }
I20260929 00:50:15.247963 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.248147 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:50:15.248178 18185 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:15.251425 18199 catalog_manager.cc:1259] Loaded cluster ID: 71fae3a7ca6b4a8eb90f5b99d5adcccb
I20260929 00:50:15.251534 18199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: loading cluster ID for follower catalog manager: success
I20260929 00:50:15.253223 18199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: acquiring CA information for follower catalog manager: success
I20260929 00:50:15.254361 18199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:50:15.340437 17739 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
W20260929 00:50:15.353967 18121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:15.354269 18121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:15.354321 18121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:15.358798 18121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:15.358973 18121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:15.359052 18121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:15.359087 18121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:15.359107 18121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:15.366418 18121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.252.keytab
--principal=oryx/127.11.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:15.367660 18121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:15.368901 18121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:15.379310 18204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.379312 18207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.379714 18121 server_base.cc:1034] running on GCE node
W20260929 00:50:15.379805 18205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.380419 18121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:15.381695 18121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:15.382908 18121 hybrid_clock.cc:648] HybridClock initialized: now 1790643015382895 us; error 30 us; skew 500 ppm
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:15.386059 18121 init.cc:377] Logged in from keytab as oryx/127.11.40.252@KRBTEST.COM (short username oryx)
I20260929 00:50:15.387061 18121 webserver.cc:460] Webserver started at http://127.11.40.252:38515/ using document root <none> and password file <none>
I20260929 00:50:15.387674 18121 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.387739 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.387912 18121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:15.390257 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/instance:
uuid: "9442bd368dad459ab2090bda3ec8558e"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "c799b4696589f9ce229d3a58eba5ae4e"
server_key_iv: "896ccb45a89ee10f581ac3ec9ee01ccb"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.390798 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal/instance:
uuid: "9442bd368dad459ab2090bda3ec8558e"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "c799b4696589f9ce229d3a58eba5ae4e"
server_key_iv: "896ccb45a89ee10f581ac3ec9ee01ccb"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.394869 18121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:15.397552 18214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.399034 18121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:15.399204 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "9442bd368dad459ab2090bda3ec8558e"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "c799b4696589f9ce229d3a58eba5ae4e"
server_key_iv: "896ccb45a89ee10f581ac3ec9ee01ccb"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.399339 18121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.430932 18121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:15.434413 18121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:15.434646 18121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:15.446338 18121 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.252:34399
I20260929 00:50:15.446632 18266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.252:34399 every 8 connection(s)
I20260929 00:50:15.447593 18121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
I20260929 00:50:15.450568 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18121
I20260929 00:50:15.451371 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:15.456861 18267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.252@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:15.467861 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.452214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.11.40.252:34399)
0929 00:50:15.452787 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.452846 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.452883 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:50:15.453678 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.453697 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.454067 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.454366 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.455520 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.456623 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.457654 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.458224 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.458422 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.461575 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.461581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.461591 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.461619 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.463868 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.464408 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.464414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.464419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.464664 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.465097 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.465102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.465104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.465687 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.465973 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.466228 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.466522 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":768,"thread_start_us":223,"threads_started":1}
I20260929 00:50:15.469983 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.458901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.11.40.254:40939)
0929 00:50:15.459066 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.459070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.459307 (+ 237us) server_negotiation.cc:407] Connection header received
0929 00:50:15.459438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.459441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.459501 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.459604 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.460608 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.461312 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.462531 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.462916 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.463043 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.466067 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.466071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.466074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.466106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.467836 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.468465 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.468468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.468469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.468556 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.468910 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.468915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.468917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.469191 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.469303 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.469694 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.469827 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:50:15.470089 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.458706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49840)
0929 00:50:15.459150 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.459202 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.459330 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.459655 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.459659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.460065 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.460447 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.460462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.461456 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.461462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.462349 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.462358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.463046 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.463051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.463169 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.463855 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.463867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.465863 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.468024 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.468036 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.468044 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.468297 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.468668 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.468673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.468675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.468769 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.469304 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.469310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.469448 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.469615 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.469813 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":8448,"thread_start_us":192,"threads_started":1}
I20260929 00:50:15.471294 18267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.252@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
I20260929 00:50:15.480504 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.471820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50226)
0929 00:50:15.471922 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.471934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.472001 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.472456 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.472459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.472624 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.472887 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.472895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.473806 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.473811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.474356 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.474364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.474842 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.474846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.475003 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.475462 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.475473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.476795 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.478791 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.478796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.478798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.479016 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.479342 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.479345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.479347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.479449 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.479923 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.479926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.480029 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.480144 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.480322 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 00:50:15.480566 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.471903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.11.40.253:39543)
0929 00:50:15.472233 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.472237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.472249 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:50:15.472285 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.472288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.472334 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.472444 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.473007 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.473685 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.474484 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.474721 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.474844 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.477010 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.477013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.477015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.477040 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.478563 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.479129 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.479135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.479139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.479232 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.479544 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.479549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.479550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.479812 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.479925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.480211 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.480351 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":146,"threads_started":1}
I20260929 00:50:15.481660 18267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.252@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
I20260929 00:50:15.490860 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.482194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.0.0.1:39436)
0929 00:50:15.482377 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.482393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.482482 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.482727 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.482729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.482918 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.482995 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.483000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.484293 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.484296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.484790 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.484798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.485304 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.485307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.485378 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.485923 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.485945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.487461 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.489402 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.489405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.489406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.489613 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.489883 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.489885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.489886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.489969 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.490397 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.490399 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.490520 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.490607 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.490724 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 00:50:15.490978 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.482280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.11.40.252:34399)
0929 00:50:15.482378 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.482383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.482438 (+ 55us) server_negotiation.cc:407] Connection header received
0929 00:50:15.482586 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.482589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.482628 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.482728 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.483113 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.483702 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.484944 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.485192 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.485304 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.487652 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.487655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.487657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.487679 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.489262 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.489713 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.489715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.489717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.489794 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.490083 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.490086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.490087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.490291 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.490397 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.490695 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.490797 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:50:15.494186 18267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Bootstrap starting.
I20260929 00:50:15.496594 18267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:15.497536 18267 log.cc:824] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:15.499778 18267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: No bootstrap required, opened a new log
I20260929 00:50:15.502943 18267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } }
I20260929 00:50:15.503209 18267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:15.503301 18267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9442bd368dad459ab2090bda3ec8558e, State: Initialized, Role: FOLLOWER
I20260929 00:50:15.503762 18267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } }
I20260929 00:50:15.504552 18273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } } }
I20260929 00:50:15.504693 18273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:15.505179 18267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:50:15.510782 18284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:50:15.510905 18284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:50:15.514076 18121 master_runner.cc:428] Detected that this master 9442bd368dad459ab2090bda3ec8558e is joining an existing cluster
I20260929 00:50:15.514158 18121 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.40.252:34399
I20260929 00:50:15.521849 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17610
I20260929 00:50:15.527508 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.516375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49856)
0929 00:50:15.517279 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.517306 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.517499 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.517893 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.517896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.518357 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.518550 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.518572 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519739 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.519743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.521035 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.521043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.521750 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.521838 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.522719 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.522740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.523290 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.525559 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.525562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.525565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.525822 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.526136 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.526139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.526141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.526223 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.526761 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.526764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.526875 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.527082 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.527239 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":122,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260929 00:50:15.527580 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.516539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.11.40.254:40939)
0929 00:50:15.516862 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.516868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.517392 (+ 524us) server_negotiation.cc:407] Connection header received
0929 00:50:15.517636 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.517639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.517692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.517744 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.518763 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519596 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521163 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.521500 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521582 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.523471 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.523475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.523478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.523508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.525432 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.525934 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.525938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.525940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.526027 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.526343 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.526350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.526355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.526651 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.526763 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.527172 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.527322 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260929 00:50:15.528184 18273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:15.528400 18273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } }
I20260929 00:50:15.529681 18273 leader_election.cc:288] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939), f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253:39543)
I20260929 00:50:15.530189 17558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9442bd368dad459ab2090bda3ec8558e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08ab8798c944c868d040a4f58fdee83" is_pre_election: true
I20260929 00:50:15.531215 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9442bd368dad459ab2090bda3ec8558e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" is_pre_election: true
I20260929 00:50:15.531420 18159 raft_consensus.cc:1799] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Handling vote request from an unknown peer 9442bd368dad459ab2090bda3ec8558e
I20260929 00:50:15.531591 18159 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9442bd368dad459ab2090bda3ec8558e for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 00:50:15.531980 18217 leader_election.cc:302] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9442bd368dad459ab2090bda3ec8558e; no voters: f08ab8798c944c868d040a4f58fdee83, f7ef0b11ae2d4ed0b3a40220fd2c32a3
I20260929 00:50:15.532415 18273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:15.533303 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:38333
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:40939
--webserver_port=34529
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:50:15.534229 18273 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643013, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.254@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
I20260929 00:50:15.547531 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.535316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.0.0.1:39454)
0929 00:50:15.535701 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.535718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.535808 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.536646 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.536650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.536945 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.537056 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.537064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.538170 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.538174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.539061 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.539072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.539769 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.539772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.539974 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.540564 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.540579 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.542262 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.545829 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.545832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.545834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.546082 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.546443 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.546446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.546448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.546537 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.546993 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.546996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.547102 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.547182 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.547342 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 00:50:15.548565 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.535973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.11.40.252:34399)
0929 00:50:15.536306 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.536316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.536350 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:50:15.536430 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.536434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.536502 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.536603 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.537185 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.538048 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.539192 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.539635 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.539764 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.542411 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.542415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.542417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.542441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.545687 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.546199 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.546202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.546205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.546333 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.546646 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.546649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.546651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.546885 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.548087 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.548163 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.548288 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260929 00:50:15.549063 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.517288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.11.40.252:34399)
0929 00:50:15.517458 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.517463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.517551 (+ 88us) server_negotiation.cc:407] Connection header received
0929 00:50:15.517702 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.517705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.517742 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.517788 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.518654 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519428 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521265 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.521619 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521694 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.542241 (+ 20547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.542246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.542251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.542281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.544424 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.547781 (+ 3357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.547785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.547786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.547877 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.548124 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.548127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.548129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.548309 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.548382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.548751 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.548902 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 00:50:15.549510 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.516846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.0.0.1:39450)
0929 00:50:15.517480 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.517493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.517610 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.517960 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.517963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.518344 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.518536 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.518545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519561 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.519564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.521144 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.521154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.521840 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.521844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.522129 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.541498 (+ 19369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.541517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.542050 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.544543 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.544549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.544554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.547674 (+ 3120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.547969 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.547972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.547974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.548045 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.548421 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.548424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.548497 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.549228 (+ 731us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.549360 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":20,"thread_start_us":312,"threads_started":1}
I20260929 00:50:15.550206 17547 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.40.252:34399: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9442bd368dad459ab2090bda3ec8558e"
member_type: NON_VOTER
last_known_addr {
host: "127.11.40.252"
port: 34399
}
attrs {
promote: true
}
}
dest_uuid: "f08ab8798c944c868d040a4f58fdee83"
cas_config_opid_index: 5
I20260929 00:50:15.550853 17547 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
I20260929 00:50:15.551687 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.516746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.11.40.253:39543)
0929 00:50:15.516870 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.516876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.517390 (+ 514us) server_negotiation.cc:407] Connection header received
0929 00:50:15.517640 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.517643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.517692 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.517744 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.518844 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519595 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.535068 (+ 15473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.535466 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.535583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.545506 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.545513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.545519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.545558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.547428 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.550346 (+ 2918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.550353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.550358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.550459 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.550762 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.550765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.550767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.550988 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.551066 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.551403 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.551536 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:50:15.551891 18108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9442bd368dad459ab2090bda3ec8558e (127.11.40.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.552143 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.516423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50234)
0929 00:50:15.517303 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.517315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.517529 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.517893 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.517896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.518391 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.518704 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.518716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.519739 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.519743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.520797 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.534928 (+ 14131us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.535871 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.535877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.536050 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.543740 (+ 7690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.543762 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.545185 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.549984 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.549987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.549989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.550216 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.550573 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.550578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.550582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.550663 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.551120 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.551122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.551225 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.551811 (+ 586us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.551972 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":110,"spinlock_wait_cycles":4480,"thread_start_us":142,"threads_started":1}
I20260929 00:50:15.553396 18241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:50:15.553934 18109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:15.554627 18108 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.40.252:34399
I20260929 00:50:15.555055 18121 master.cc:537] Master@127.11.40.252:34399 shutting down...
I20260929 00:50:15.558140 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.558308 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:15.558460 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.552067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50250)
0929 00:50:15.552917 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.552931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.553009 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.554123 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.554126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.554144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:15.554263 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.554271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.555278 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.555281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.556582 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.556597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.557905 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.557977 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.558102 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.558144 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.558226 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.558320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771}
I20260929 00:50:15.558988 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.553319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.11.40.253:39543)
0929 00:50:15.553776 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.553780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.553792 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:50:15.553840 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.553842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.553906 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.554003 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:15.554395 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.555148 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.556732 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.557485 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.557603 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.557814 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.558688 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.558876 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260929 00:50:15.559542 18241 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
W20260929 00:50:15.559666 18241 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
I20260929 00:50:15.560169 18159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:50:15.560640 18109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 00:50:15.561336 18241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.562917 18159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9442bd368dad459ab2090bda3ec8558e (127.11.40.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.563704 18273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ab8798c944c868d040a4f58fdee83. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.563805 18273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 00:50:15.563895 18273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.563943 18273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 00:50:15.563997 18273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } }
I20260929 00:50:15.564049 18273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 00:50:15.565228 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ab8798c944c868d040a4f58fdee83. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.565400 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:50:15.565510 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.565605 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:50:15.566773 18296 mvcc.cc:205] Tried to move back new op lower bound from 7334473785647140864 to 7334473785463103488. Current Snapshot: MvccSnapshot[applied={T|T < 7334473785501044736}]
I20260929 00:50:15.571482 18295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: NON_VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.571666 18295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:15.572132 18108 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: attempting to promote NON_VOTER f7ef0b11ae2d4ed0b3a40220fd2c32a3 to VOTER
I20260929 00:50:15.572580 18108 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
I20260929 00:50:15.574069 18159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 LEARNER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 00:50:15.574297 17532 consensus_peers.cc:487] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 -> Peer 9442bd368dad459ab2090bda3ec8558e (127.11.40.252:34399): Couldn't send request to peer 9442bd368dad459ab2090bda3ec8558e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:50:15.574922 18295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 00:50:15.575019 18121 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 00:50:15.575325 18121 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 00:50:15.575415 18121 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: stopping tablet replica
I20260929 00:50:15.577003 18295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.578765 18295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.578897 18295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:15.579258 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.579422 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:15.580384 18159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.583019 18301 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.583591 18301 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:15.595692 18121 master.cc:552] Master@127.11.40.252:34399 shutdown complete.
I20260929 00:50:15.595778 18121 master_runner.cc:305] Clearing existing system tablet
I20260929 00:50:15.596066 18121 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.596136 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.597645 18121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 00:50:15.598791 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.599160 18121 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.599305 18121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:50:15.599457 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "9442bd368dad459ab2090bda3ec8558e"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "c799b4696589f9ce229d3a58eba5ae4e"
server_key_iv: "896ccb45a89ee10f581ac3ec9ee01ccb"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.601516 18121 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:15.601933 17873 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
I20260929 00:50:15.604682 18121 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:50:15.604796 18121 log.cc:1196] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 00:50:15.605159 18121 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Deleting consensus metadata
I20260929 00:50:15.605352 18121 master_runner.cc:315] Copying system tablet from 127.11.40.254:40939
I20260929 00:50:15.606621 18121 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.40.254:40939
I20260929 00:50:15.618597 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.607494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:49870)
0929 00:50:15.608098 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.608181 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.608340 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.608680 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.608683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.609068 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.609644 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.609656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.610864 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.610868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.611792 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.611808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.612462 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.612465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.612551 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.613241 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.613259 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.613729 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.616093 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.616099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.616103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.617045 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.617409 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.617415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.617418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.617533 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.618025 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.618029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.618144 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.618246 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.618407 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":97,"threads_started":1}
I20260929 00:50:15.618950 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.607646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.11.40.254:40939)
0929 00:50:15.607779 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.607786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.608311 (+ 525us) server_negotiation.cc:407] Connection header received
0929 00:50:15.608478 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.608482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.608541 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.608634 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.609805 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.610708 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.611954 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.612320 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.612437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.614030 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.614034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.614038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.614071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.615954 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.617172 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.617176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.617178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.617286 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.617652 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.617655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.617657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.617902 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.618010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.618346 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.618482 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:50:15.619318 17568 tablet_copy_service.cc:140] P f08ab8798c944c868d040a4f58fdee83: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9442bd368dad459ab2090bda3ec8558e ({username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:49870)
I20260929 00:50:15.619421 17568 tablet_copy_service.cc:161] P f08ab8798c944c868d040a4f58fdee83: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9442bd368dad459ab2090bda3ec8558e at {username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:49870: session id = 9442bd368dad459ab2090bda3ec8558e-00000000000000000000000000000000
I20260929 00:50:15.620559 17568 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:50:15.621939 18121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:50:15.625216 18121 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: tablet copy: Starting download of 0 data blocks...
I20260929 00:50:15.625425 18121 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: tablet copy: Starting download of 1 WAL segments...
I20260929 00:50:15.627310 18121 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:50:15.629571 17568 tablet_copy_service.cc:342] P f08ab8798c944c868d040a4f58fdee83: Request end of tablet copy session 9442bd368dad459ab2090bda3ec8558e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:49870
I20260929 00:50:15.629678 17568 tablet_copy_service.cc:433] P f08ab8798c944c868d040a4f58fdee83: ending tablet copy session 9442bd368dad459ab2090bda3ec8558e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9442bd368dad459ab2090bda3ec8558e
I20260929 00:50:15.637708 18121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:15.639982 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.642910 18121 server_base.cc:1034] running on GCE node
W20260929 00:50:15.643071 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.645031 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.645574 18121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:15.646275 18121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:15.647455 18121 hybrid_clock.cc:648] HybridClock initialized: now 1790643015647433 us; error 48 us; skew 500 ppm
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:15.649479 18121 init.cc:377] Logged in from keytab as oryx/127.11.40.252@KRBTEST.COM (short username oryx)
I20260929 00:50:15.650048 18121 webserver.cc:460] Webserver started at http://127.11.40.252:45033/ using document root <none> and password file <none>
I20260929 00:50:15.650269 18121 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.650403 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.651981 18121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:15.653214 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.653553 18121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:15.653633 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "9442bd368dad459ab2090bda3ec8558e"
format_stamp: "Formatted at 2026-09-29 00:50:15 on dist-test-slave-5309"
server_key: "c799b4696589f9ce229d3a58eba5ae4e"
server_key_iv: "896ccb45a89ee10f581ac3ec9ee01ccb"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.653720 18121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.672468 18121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:15.675042 18121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:15.681454 18121 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.252:34399
I20260929 00:50:15.681488 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.252:34399 every 8 connection(s)
I20260929 00:50:15.682096 18121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
I20260929 00:50:15.685154 18377 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:50:15.686632 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Bootstrap starting.
I20260929 00:50:15.697854 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:50:15.698256 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Bootstrap complete.
I20260929 00:50:15.698936 18377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
I20260929 00:50:15.699088 18377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9442bd368dad459ab2090bda3ec8558e, State: Initialized, Role: LEARNER
I20260929 00:50:15.699211 18377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } }
I20260929 00:50:15.699692 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:15.699812 18377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:15.699937 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: LEARNER
I20260929 00:50:15.705128 18391 catalog_manager.cc:1259] Loaded cluster ID: 71fae3a7ca6b4a8eb90f5b99d5adcccb
I20260929 00:50:15.705215 18391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: loading cluster ID for follower catalog manager: success
I20260929 00:50:15.707683 18391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: acquiring CA information for follower catalog manager: success
I20260929 00:50:15.709396 18391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 00:50:15.723066 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:15.723398 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:15.723446 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:15.728521 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:15.728629 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:15.728694 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:15.735072 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:38333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=34529
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:15.736375 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:15.737617 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:15.750355 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.750272 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:15.750434 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:15.751178 18293 server_base.cc:1034] running on GCE node
I20260929 00:50:15.751806 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:15.752663 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:15.753918 18293 hybrid_clock.cc:648] HybridClock initialized: now 1790643015753896 us; error 38 us; skew 500 ppm
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:15.757450 18293 init.cc:377] Logged in from keytab as oryx/127.11.40.193@KRBTEST.COM (short username oryx)
I20260929 00:50:15.758525 18293 webserver.cc:460] Webserver started at http://127.11.40.193:34529/ using document root <none> and password file <none>
I20260929 00:50:15.759215 18293 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:15.759348 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:15.763993 18293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:15.766638 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.768249 18293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:50:15.768524 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "ad357811cbfb0e2db96b4b18e8fa6ac1"
server_key_iv: "30aa376eb135be05ca51b458e4026500"
server_key_version: "encryptionkey@0"
I20260929 00:50:15.769297 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:15.795524 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:15.800137 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:15.800385 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:15.801326 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:15.802479 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:15.802554 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.802623 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:15.802673 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:15.820012 18293 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:38333
I20260929 00:50:15.820159 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:38333 every 8 connection(s)
I20260929 00:50:15.821544 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:15.823873 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18293
I20260929 00:50:15.824137 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17744
I20260929 00:50:15.832888 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:35963
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:40939
--webserver_port=36289
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:50:15.835227 18007 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
Sep 29 00:50:15 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643015, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
I20260929 00:50:15.844977 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.825629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37709 (local address 127.11.40.253:39543)
0929 00:50:15.825815 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.825822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.827731 (+ 1909us) server_negotiation.cc:407] Connection header received
0929 00:50:15.828969 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.828972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.829050 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.829103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.830953 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.831811 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.836970 (+ 5159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.837514 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.837653 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.841153 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.841160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.841166 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.841203 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.842974 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843479 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.843483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.843484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.843566 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843871 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.843875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.843879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.844086 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.844170 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.844615 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.844760 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:50:15.844987 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.824169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:42283 (local address 127.11.40.252:34399)
0929 00:50:15.824538 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.824546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.826348 (+ 1802us) server_negotiation.cc:407] Connection header received
0929 00:50:15.828472 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.828480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.828557 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.828630 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.830957 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.832155 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.834411 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.834809 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.834938 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.839682 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.839688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.839693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.839736 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.841921 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.842869 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.842873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.842875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.842972 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843476 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.843480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.843482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.843751 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.843842 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.844615 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.844761 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260929 00:50:15.845521 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.826479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:42297 (local address 127.11.40.254:40939)
0929 00:50:15.826672 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:15.826680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:15.827161 (+ 481us) server_negotiation.cc:407] Connection header received
0929 00:50:15.828527 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:15.828533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:15.828602 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:15.828705 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:15.830954 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.831811 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.835139 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.835539 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.835631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.839690 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:15.839697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:15.839703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:15.839746 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:15.841948 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.842881 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.842887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.842892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.843122 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843569 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:15.843576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:15.843579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:15.844163 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:15.844290 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:15.844727 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 00:50:15.844938 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:50:15.847456 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.824894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.11.40.193:42283)
0929 00:50:15.825612 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.825670 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.828162 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.828977 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.829000 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.829719 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.830754 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.830781 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.832326 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.832333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.834103 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.834121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.835154 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.835158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.835456 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.836231 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.836253 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.839417 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.842090 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.842101 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.842128 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.842721 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.843094 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.843099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.843329 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.843911 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.843922 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.844306 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.845366 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.845964 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":238,"spinlock_wait_cycles":20480,"thread_start_us":245,"threads_started":1}
I20260929 00:50:15.847807 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.824894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.11.40.193:37709)
0929 00:50:15.827612 (+ 2718us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.827629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.828811 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.829311 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.829314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.829721 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.830759 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.830779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.832027 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.832031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.834969 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.834980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.837880 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.837884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.838017 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.838601 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.838620 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.840949 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.843099 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.843104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.843355 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.843675 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.843679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.843756 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.844225 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.844227 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.844306 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.845426 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.846027 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2546,"mutex_wait_us":3504,"spinlock_wait_cycles":7552,"thread_start_us":259,"threads_started":1}
I20260929 00:50:15.847456 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:15.826219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:42297)
0929 00:50:15.827027 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:15.827045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:15.828168 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:15.828965 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:15.828977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:15.829721 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:15.830757 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.830779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.832027 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.832031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.834956 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:15.834978 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:15.835824 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:15.835828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:15.835933 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:15.836511 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:15.836532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:15.839416 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:15.842123 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.842128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.842133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.842725 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.843293 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:15.843297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:15.843301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:15.843434 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:15.844366 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:15.844369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:15.844492 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 00:50:15.845735 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0929 00:50:15.846149 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":1746,"spinlock_wait_cycles":8704,"thread_start_us":1064,"threads_started":1}
I20260929 00:50:15.849763 18522 heartbeater.cc:347] Connected to a master server at 127.11.40.253:39543
I20260929 00:50:15.849779 18521 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:15.849788 18520 heartbeater.cc:347] Connected to a master server at 127.11.40.252:34399
I20260929 00:50:15.850071 18522 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:15.850071 18521 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:15.850764 18521 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
I20260929 00:50:15.850821 18522 heartbeater.cc:511] Master 127.11.40.253:39543 requested a full tablet report, sending...
I20260929 00:50:15.850071 18520 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:15.851603 18520 heartbeater.cc:511] Master 127.11.40.252:34399 requested a full tablet report, sending...
I20260929 00:50:15.852334 17547 ts_manager.cc:194] Re-registered known tserver with Master: e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193:38333)
I20260929 00:50:15.852535 18145 ts_manager.cc:194] Registered new tserver with Master: e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193:38333)
I20260929 00:50:15.853271 17547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:42297
I20260929 00:50:15.853924 18340 ts_manager.cc:194] Registered new tserver with Master: e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193:38333)
W20260929 00:50:15.973448 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:15.973809 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:15.973884 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:15.978812 18528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:15.978984 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:15.979094 18528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:15.987282 18528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:35963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=36289
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:15.988600 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:15.989867 18528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:16.002283 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:16.002283 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:16.002290 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:16.002885 18528 server_base.cc:1034] running on GCE node
I20260929 00:50:16.003464 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:16.004285 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:16.007718 18528 hybrid_clock.cc:648] HybridClock initialized: now 1790643016007490 us; error 240 us; skew 500 ppm
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:16.011876 18528 init.cc:377] Logged in from keytab as oryx/127.11.40.194@KRBTEST.COM (short username oryx)
I20260929 00:50:16.013384 18528 webserver.cc:460] Webserver started at http://127.11.40.194:36289/ using document root <none> and password file <none>
I20260929 00:50:16.014297 18528 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:16.014422 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:16.019905 18528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:50:16.020066 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.016104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.11.40.252:34399)
0929 00:50:16.016254 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.016258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.016396 (+ 138us) server_negotiation.cc:407] Connection header received
0929 00:50:16.016583 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.016586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.016641 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.016745 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.017045 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.017604 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.018786 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.019454 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.019587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.019764 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.019842 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.019932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:50:16.020094 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.016117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.0.0.1:39464)
0929 00:50:16.016298 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.016314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.016455 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.016767 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.016771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.016788 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.016911 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.016917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.017732 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.017736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.018606 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.018619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.019639 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.019642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.019723 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.019746 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.019820 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.019930 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 00:50:16.020881 18351 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 00:50:16.021179 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ab8798c944c868d040a4f58fdee83. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:16.021368 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: LEARNER
I20260929 00:50:16.023109 18543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.025360 18528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:50:16.025529 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "312dcaf687e64010a9db6a66b73f068f"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "e204d23e7c943b892d0f8250d69b517d"
server_key_iv: "f96a20738d4dbd255db9e7a5bf39e4b6"
server_key_version: "encryptionkey@0"
I20260929 00:50:16.026173 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:16.041800 18528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:16.047930 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:16.048273 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:16.049351 18528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:16.050562 18528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:16.050616 18528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.050659 18528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:16.050675 18528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.065121 18528 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:35963
I20260929 00:50:16.065307 18656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:35963 every 8 connection(s)
I20260929 00:50:16.066428 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:16.073777 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18528
I20260929 00:50:16.073984 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17878
I20260929 00:50:16.082511 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:43241
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:40939
--webserver_port=41251
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
I20260929 00:50:16.090843 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.068978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:40885 (local address 127.11.40.254:40939)
0929 00:50:16.069151 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.069155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.071493 (+ 2338us) server_negotiation.cc:407] Connection header received
0929 00:50:16.073085 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.073092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.073166 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.073226 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.075728 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.076289 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078153 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078518 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078610 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.084464 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.084469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.084472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.084509 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.087661 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.088315 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.088319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.088321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.088415 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.089126 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.089130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.089134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.089361 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.089481 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.090438 (+ 957us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.090606 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:50:16.092492 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.070077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:40885)
0929 00:50:16.071166 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.071244 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.072493 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.073506 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.073518 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.075334 (+ 1816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.075558 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.075570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.076544 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.076548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.078004 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.078014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078948 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.079315 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.080540 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.080567 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.084219 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.087827 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.087836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.087859 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.088174 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.088723 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.088725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.088727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.089002 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.089481 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.089492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.089807 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.090361 (+ 554us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.090798 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":131,"spinlock_wait_cycles":38912,"thread_start_us":255,"threads_started":1}
I20260929 00:50:16.092720 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.070077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.11.40.194:59757)
0929 00:50:16.071166 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.071244 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.072493 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.077106 (+ 4613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.077110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.077471 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.077598 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.077605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078707 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.081092 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.081108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.082229 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.082239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.082420 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.083163 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.083181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.086470 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.089706 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.089712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.089716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.090059 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.090463 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.090466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.090468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.090581 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.091517 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.091522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.091647 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.091771 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.091960 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"spinlock_wait_cycles":8448,"thread_start_us":199,"threads_started":1}
I20260929 00:50:16.093324 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.069563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:59757 (local address 127.11.40.253:39543)
0929 00:50:16.069695 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.069699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.071427 (+ 1728us) server_negotiation.cc:407] Connection header received
0929 00:50:16.073176 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.073180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.073238 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.073323 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.077780 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078570 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.081315 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.081742 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.081885 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.086708 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.086715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.086721 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.086762 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.089547 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.090216 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.090222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.090227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.090332 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.090716 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.090722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.090725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.091343 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.091460 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.092435 (+ 975us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.093137 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:50:16.094316 18659 heartbeater.cc:347] Connected to a master server at 127.11.40.253:39543
I20260929 00:50:16.094769 18659 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:16.095340 18659 heartbeater.cc:511] Master 127.11.40.253:39543 requested a full tablet report, sending...
I20260929 00:50:16.095865 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:60111 (local address 127.11.40.252:34399)
0929 00:50:16.071823 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.071828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.072164 (+ 336us) server_negotiation.cc:407] Connection header received
0929 00:50:16.073026 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.073033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.073095 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.073195 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.075121 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.075709 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078163 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078611 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078715 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.084441 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.084449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.084453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.084495 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.087959 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.094097 (+ 6138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.094105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.094110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.094245 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.094631 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.094637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.094640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.094918 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.095037 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.095458 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.095634 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:50:16.096041 18658 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:16.096133 18658 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:16.096447 18658 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
I20260929 00:50:16.096491 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.11.40.194:60111)
0929 00:50:16.072017 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.072033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.072535 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.073561 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.073563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.074162 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.074956 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.074975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.075971 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.075978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.078005 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.078020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.078948 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.078954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.079318 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.080773 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.080794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.084146 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.088548 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.088554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.088559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.093916 (+ 5357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.094388 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.094394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.094397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.094500 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.095108 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.095113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.095236 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.096038 (+ 802us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.096235 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":112,"thread_start_us":157,"threads_started":1}
I20260929 00:50:16.097517 18145 ts_manager.cc:194] Registered new tserver with Master: 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
I20260929 00:50:16.098151 17547 ts_manager.cc:194] Re-registered known tserver with Master: 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
I20260929 00:50:16.098171 18657 heartbeater.cc:347] Connected to a master server at 127.11.40.252:34399
I20260929 00:50:16.098591 18657 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:16.098814 18657 heartbeater.cc:511] Master 127.11.40.252:34399 requested a full tablet report, sending...
I20260929 00:50:16.099185 17547 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:40885
I20260929 00:50:16.100107 18340 ts_manager.cc:194] Registered new tserver with Master: 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
W20260929 00:50:16.221783 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:16.222085 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:16.222131 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:16.226485 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:16.226574 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:16.226639 18664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:16.232785 18664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:43241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=41251
--tserver_master_addrs=127.11.40.254:40939,127.11.40.253:39543,127.11.40.252:34399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:16.234135 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:16.235306 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:16.246296 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:16.246310 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:16.246512 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:16.247268 18664 server_base.cc:1034] running on GCE node
I20260929 00:50:16.247884 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:16.248852 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:16.250073 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790643016250033 us; error 58 us; skew 500 ppm
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:16.253489 18664 init.cc:377] Logged in from keytab as oryx/127.11.40.195@KRBTEST.COM (short username oryx)
I20260929 00:50:16.254654 18664 webserver.cc:460] Webserver started at http://127.11.40.195:41251/ using document root <none> and password file <none>
I20260929 00:50:16.255292 18664 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:16.255424 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:16.260105 18664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:16.262730 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.264489 18664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:16.264673 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "47a69599c9e242d19b026d29b96d3e7b"
format_stamp: "Formatted at 2026-09-29 00:50:14 on dist-test-slave-5309"
server_key: "568826de0c4aa4b7dfbe8d2bd00cde61"
server_key_iv: "518c76caa3b69c6dd5d0f4e06ab5c95a"
server_key_version: "encryptionkey@0"
I20260929 00:50:16.265664 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:16.281895 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:16.285823 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:16.286067 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:16.286803 18664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:16.287772 18664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:16.287846 18664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.287919 18664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:16.287973 18664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:16.298677 18664 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:43241
I20260929 00:50:16.298749 18792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:43241 every 8 connection(s)
I20260929 00:50:16.299929 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:16.302172 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18664
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.253@KRBTEST.COM
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.252@KRBTEST.COM
I20260929 00:50:16.320433 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.302723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:56059 (local address 127.11.40.253:39543)
0929 00:50:16.303066 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.303071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.303958 (+ 887us) server_negotiation.cc:407] Connection header received
0929 00:50:16.305336 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.305340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.305401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.305480 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.307759 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.308511 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.310387 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.310727 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.310823 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.314033 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.314037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.314040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.314068 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.316493 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319071 (+ 2578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319169 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319496 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319696 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.319784 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.320143 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.320259 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260929 00:50:16.320468 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.302222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:51351 (local address 127.11.40.252:34399)
0929 00:50:16.302522 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.302527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.304704 (+ 2177us) server_negotiation.cc:407] Connection header received
0929 00:50:16.305333 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.305340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.305407 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.305467 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.307484 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.308379 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.311109 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.311421 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.311512 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.315768 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.315772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.315777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.315810 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.317723 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.318477 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.318482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.318487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.318586 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319036 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319291 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.319368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.320121 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.320296 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":203}
I20260929 00:50:16.320776 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.305057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.11.40.252:34399)
0929 00:50:16.306214 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.306218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.306230 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:50:16.306264 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.306267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.306320 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.306406 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.307506 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.310768 (+ 3262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.313804 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.314310 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.314439 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.316192 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.316199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.316205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.316235 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.318807 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319297 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319377 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319666 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319870 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.319986 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.320514 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.320628 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2453,"server-negotiator.queue_time_us":1076,"thread_start_us":71,"threads_started":1}
I20260929 00:50:16.321357 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.305276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.11.40.253:39543)
0929 00:50:16.306286 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.306290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.307343 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 00:50:16.307514 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.307518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.307564 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.307628 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.308575 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.309462 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.310672 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.311066 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.311146 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.313042 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.313046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.313049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.313076 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.317389 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.317978 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.317982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.317985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.318079 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.318413 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.318419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.318422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.318679 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.318786 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.321083 (+ 2297us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.321198 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":62,"threads_started":1}
I20260929 00:50:16.322489 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.305034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.11.40.254:40939)
0929 00:50:16.306426 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.306430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.306442 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:50:16.306479 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.306482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.306528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.306969 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.307665 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.308544 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.309762 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.310115 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.310215 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.313042 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.313046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.313049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.313076 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.318210 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319104 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319197 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319515 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.319518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.319519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.319717 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.319843 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.320468 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.321195 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":74,"threads_started":1}
Sep 29 00:50:16 dist-test-slave-5309 krb5kdc[17503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643016, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:16.323371 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.302906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.11.40.195:51351)
0929 00:50:16.304610 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.304625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.305083 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.305904 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.305913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.306503 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.307308 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.307329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.308564 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.308567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.310987 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.310994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.311698 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.311704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.311803 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.313079 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.313094 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.315592 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.317866 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.317875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.317897 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.318350 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.318700 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.318703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.318705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.318927 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.319460 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.319469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.319839 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.320680 (+ 841us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.322023 (+ 1343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"mutex_wait_us":1065,"spinlock_wait_cycles":43136,"thread_start_us":202,"threads_started":1}
I20260929 00:50:16.323693 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.302906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.11.40.195:56059)
0929 00:50:16.303773 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.303836 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.305077 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.305906 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.305913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.306586 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.307637 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.307644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.308638 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.308641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.310253 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.310263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.311026 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.311033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.311255 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.311831 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.311854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.313820 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.318689 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.318692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.318694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.318975 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.319280 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.319283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.319286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.319380 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.319857 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.319861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.319953 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.320847 (+ 894us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.322077 (+ 1230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1798,"spinlock_wait_cycles":5248,"thread_start_us":165,"threads_started":1}
I20260929 00:50:16.325215 18795 heartbeater.cc:347] Connected to a master server at 127.11.40.253:39543
I20260929 00:50:16.325418 18793 heartbeater.cc:347] Connected to a master server at 127.11.40.252:34399
I20260929 00:50:16.325490 18795 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:16.325537 18793 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:16.326149 18795 heartbeater.cc:511] Master 127.11.40.253:39543 requested a full tablet report, sending...
I20260929 00:50:16.326150 18793 heartbeater.cc:511] Master 127.11.40.252:34399 requested a full tablet report, sending...
I20260929 00:50:16.327435 18340 ts_manager.cc:194] Registered new tserver with Master: 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:16.327705 18145 ts_manager.cc:194] Registered new tserver with Master: 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:16.326093 17547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:50:16.329339 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.304956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:53499 (local address 127.11.40.254:40939)
0929 00:50:16.305226 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.305232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.305250 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:50:16.305846 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.305849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.305903 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.305954 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.314297 (+ 8343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.315036 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.316422 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.316784 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.316911 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.324636 (+ 7725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.324641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.324644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.324672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.327238 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.327903 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.327909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.327914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.328008 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.328345 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.328348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.328350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.328542 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.328654 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.329026 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.329139 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260929 00:50:16.329485 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.304292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:53499)
0929 00:50:16.304610 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.304625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.305705 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.306093 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.306099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.308925 (+ 2826us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:16.313651 (+ 4726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.313664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.315189 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.315193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.316249 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.316261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.317097 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.317100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.317215 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.321576 (+ 4361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:16.321591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:16.324460 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:16.327381 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.327387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.327393 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.327712 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.328118 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:16.328121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:16.328123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:16.328216 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:16.328642 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:16.328645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:16.328746 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.328946 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.329132 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260929 00:50:16.330397 18794 heartbeater.cc:347] Connected to a master server at 127.11.40.254:40939
I20260929 00:50:16.330480 18794 heartbeater.cc:464] Registering TS with master...
W20260929 00:50:16.330467 17547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:16.330730 18794 heartbeater.cc:511] Master 127.11.40.254:40939 requested a full tablet report, sending...
I20260929 00:50:16.331413 17548 ts_manager.cc:194] Re-registered known tserver with Master: 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:16.332226 17548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:53499
I20260929 00:50:16.333642 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: NON_VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: true } } }
I20260929 00:50:16.333789 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:16.335592 18812 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: attempting to promote NON_VOTER 9442bd368dad459ab2090bda3ec8558e to VOTER
I20260929 00:50:16.335959 18812 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } }
I20260929 00:50:16.337170 18159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 00:50:16.337193 18351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 LEARNER]: Refusing update from remote peer f08ab8798c944c868d040a4f58fdee83: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 00:50:16.337528 18305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 00:50:16.337677 18811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 00:50:16.339753 18305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9442bd368dad459ab2090bda3ec8558e (127.11.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.340166 18159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9442bd368dad459ab2090bda3ec8558e (127.11.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.341601 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.341704 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:16.341842 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.341933 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:16.342025 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.342094 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:16.342510 18351 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9442bd368dad459ab2090bda3ec8558e (127.11.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.342955 18820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.343082 18820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:16.344141 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08ab8798c944c868d040a4f58fdee83" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:16.344270 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:16.354678 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.348793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35963 (local address 127.0.0.1:49630)
0929 00:50:16.349662 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.349678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.349880 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.350356 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.350360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.350390 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.350547 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.350558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.351572 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.351577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.352706 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.352716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.354070 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.354074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.354166 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.354193 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.354274 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.354415 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":24,"spinlock_wait_cycles":3456,"thread_start_us":104,"threads_started":1}
I20260929 00:50:16.354936 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.348258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38333 (local address 127.0.0.1:43970)
0929 00:50:16.348728 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.348744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.349722 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.350307 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.350311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.350328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.350518 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.350525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.352065 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.352069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.353209 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.353219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.354268 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.354272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.354359 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.354379 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.354585 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.354673 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":4,"spinlock_wait_cycles":7168}
I20260929 00:50:16.355083 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.349243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.11.40.194:35963)
0929 00:50:16.349656 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.349670 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.349851 (+ 181us) server_negotiation.cc:407] Connection header received
0929 00:50:16.349997 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.350002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.350228 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.350402 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.350682 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.351453 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.352869 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.353931 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.354037 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.354208 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.354316 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.354527 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":201,"threads_started":1}
I20260929 00:50:16.355273 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.348501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:43241 (local address 127.0.0.1:55622)
0929 00:50:16.348838 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.348853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.348971 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.350471 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.350473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.350492 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.350580 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.350586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.352450 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.352454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.353461 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.353471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.354599 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.354602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.354728 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.354759 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.354816 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.354896 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":4,"thread_start_us":177,"threads_started":1}
I20260929 00:50:16.355751 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.349407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.11.40.193:38333)
0929 00:50:16.349671 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.349680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.349706 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:50:16.349853 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.349859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.350153 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.350679 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.350805 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.351940 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.353346 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.354132 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.355166 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.355379 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.355439 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.355596 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260929 00:50:16.356215 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.348976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.11.40.195:43241)
0929 00:50:16.350051 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.350062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.350084 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:50:16.350155 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.350160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.350329 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.350700 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.350826 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.352331 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.353650 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.354464 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.355759 (+ 1295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.355903 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.355964 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.356061 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":117,"threads_started":1}
I20260929 00:50:16.358780 18727 tablet_service.cc:1467] Processing CreateTablet for tablet 8adf8a2e7b0748689221cf837ac130dc (DEFAULT_TABLE table=test-table [id=1a1cd43c970640429664d4a01960696d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:16.358798 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 8adf8a2e7b0748689221cf837ac130dc (DEFAULT_TABLE table=test-table [id=1a1cd43c970640429664d4a01960696d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:16.358767 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 8adf8a2e7b0748689221cf837ac130dc (DEFAULT_TABLE table=test-table [id=1a1cd43c970640429664d4a01960696d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:16.359907 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adf8a2e7b0748689221cf837ac130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:16.359907 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adf8a2e7b0748689221cf837ac130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:16.359932 18727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adf8a2e7b0748689221cf837ac130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:16.367672 18826 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Bootstrap starting.
I20260929 00:50:16.369283 18827 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Bootstrap starting.
I20260929 00:50:16.369283 18828 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Bootstrap starting.
I20260929 00:50:16.370292 18826 tablet_bootstrap.cc:654] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:16.372028 18826 log.cc:824] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:16.372203 18827 tablet_bootstrap.cc:654] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:16.372221 18828 tablet_bootstrap.cc:654] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:16.373509 18827 log.cc:824] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:16.373556 18828 log.cc:824] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:16.374655 18826 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: No bootstrap required, opened a new log
I20260929 00:50:16.374909 18826 ts_tablet_manager.cc:1397] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 00:50:16.375897 18828 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: No bootstrap required, opened a new log
I20260929 00:50:16.376112 18828 ts_tablet_manager.cc:1397] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 00:50:16.376430 18827 tablet_bootstrap.cc:492] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: No bootstrap required, opened a new log
I20260929 00:50:16.376629 18827 ts_tablet_manager.cc:1397] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 00:50:16.380338 18826 raft_consensus.cc:348] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.380477 18828 raft_consensus.cc:348] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.380679 18826 raft_consensus.cc:374] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:16.380714 18828 raft_consensus.cc:374] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:16.380740 18826 raft_consensus.cc:729] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a69599c9e242d19b026d29b96d3e7b, State: Initialized, Role: FOLLOWER
I20260929 00:50:16.380772 18828 raft_consensus.cc:729] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312dcaf687e64010a9db6a66b73f068f, State: Initialized, Role: FOLLOWER
I20260929 00:50:16.381268 18828 consensus_queue.cc:260] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.381260 18826 consensus_queue.cc:260] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.381174 18827 raft_consensus.cc:348] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.381417 18827 raft_consensus.cc:374] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:16.381479 18827 raft_consensus.cc:729] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9f23c5c9e4cfdaa794a786b3d9164, State: Initialized, Role: FOLLOWER
I20260929 00:50:16.381915 18827 consensus_queue.cc:260] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.382403 18826 ts_tablet_manager.cc:1428] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260929 00:50:16.382623 18794 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
I20260929 00:50:16.382809 18827 ts_tablet_manager.cc:1428] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:50:16.383608 18828 ts_tablet_manager.cc:1428] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 00:50:16.383935 18521 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
I20260929 00:50:16.387648 18658 heartbeater.cc:503] Master 127.11.40.254:40939 was elected leader, sending a full tablet report...
I20260929 00:50:16.433308 18834 raft_consensus.cc:482] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:16.433581 18834 raft_consensus.cc:504] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.434832 18834 leader_election.cc:288] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241), 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
I20260929 00:50:16.439121 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.435289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:43007 (local address 127.11.40.195:43241)
0929 00:50:16.435429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.435434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.435831 (+ 397us) server_negotiation.cc:407] Connection header received
0929 00:50:16.435980 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.435983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.436031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.436116 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.436411 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.436946 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438131 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.438529 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438688 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.438748 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.438921 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.438993 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:50:16.439143 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.435149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:43241 (local address 127.11.40.193:43007)
0929 00:50:16.435723 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.435739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.435864 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.436130 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.436132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.436148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.436272 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.436278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.437061 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.437065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.438005 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.438014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.438677 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.438777 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.438800 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.438907 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.438980 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":344,"thread_start_us":119,"threads_started":1}
I20260929 00:50:16.439193 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.435261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:60881 (local address 127.11.40.194:35963)
0929 00:50:16.435461 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.435466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.435658 (+ 192us) server_negotiation.cc:407] Connection header received
0929 00:50:16.435854 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.435857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.435914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.436011 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:16.436302 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.436892 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438132 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.438587 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438731 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.438781 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.439007 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.439084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 00:50:16.439620 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.435149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35963 (local address 127.11.40.193:60881)
0929 00:50:16.435548 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:16.435561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:16.435720 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:16.436024 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:16.436026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:16.436043 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:16.436183 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.436188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.437011 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.437016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.437998 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:16.438008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.438724 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.438728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:16.438874 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.438899 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:50:16.438989 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:50:16.439061 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":90,"thread_start_us":153,"threads_started":1}
I20260929 00:50:16.439869 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adf8a2e7b0748689221cf837ac130dc" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a69599c9e242d19b026d29b96d3e7b" is_pre_election: true
I20260929 00:50:16.440243 18747 raft_consensus.cc:2454] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 0.
I20260929 00:50:16.440313 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adf8a2e7b0748689221cf837ac130dc" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312dcaf687e64010a9db6a66b73f068f" is_pre_election: true
I20260929 00:50:16.440636 18611 raft_consensus.cc:2454] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 0.
I20260929 00:50:16.440788 18410 leader_election.cc:302] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a69599c9e242d19b026d29b96d3e7b, e8f9f23c5c9e4cfdaa794a786b3d9164; no voters:
I20260929 00:50:16.441155 18834 raft_consensus.cc:2781] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:16.441250 18834 raft_consensus.cc:482] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:16.441361 18834 raft_consensus.cc:3037] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:16.442469 18834 raft_consensus.cc:504] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.442812 18834 leader_election.cc:288] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 election: Requested vote from peers 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241), 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963)
I20260929 00:50:16.443194 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adf8a2e7b0748689221cf837ac130dc" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a69599c9e242d19b026d29b96d3e7b"
I20260929 00:50:16.443212 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adf8a2e7b0748689221cf837ac130dc" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312dcaf687e64010a9db6a66b73f068f"
I20260929 00:50:16.443343 18747 raft_consensus.cc:3037] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:16.443397 18611 raft_consensus.cc:3037] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:16.444433 18747 raft_consensus.cc:2454] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 1.
I20260929 00:50:16.444464 18611 raft_consensus.cc:2454] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 1.
I20260929 00:50:16.444720 18410 leader_election.cc:302] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a69599c9e242d19b026d29b96d3e7b, e8f9f23c5c9e4cfdaa794a786b3d9164; no voters:
I20260929 00:50:16.444933 18834 raft_consensus.cc:2781] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:16.445171 18834 raft_consensus.cc:686] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 LEADER]: Becoming Leader. State: Replica: e8f9f23c5c9e4cfdaa794a786b3d9164, State: Running, Role: LEADER
I20260929 00:50:16.445545 18834 consensus_queue.cc:237] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } }
I20260929 00:50:16.449527 17547 catalog_manager.cc:5495] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } health_report { overall_health: UNKNOWN } } }
I20260929 00:50:16.480515 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.476888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.11.40.193:38333)
0929 00:50:16.477040 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.477044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.477058 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:16.477140 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.477142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.477199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.477301 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:16.477584 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.478134 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.478971 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.479202 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.479285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.479567 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:16.480026 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:16.480123 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.480245 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.480350 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:50:16.487075 18611 raft_consensus.cc:1261] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Refusing update from remote peer e8f9f23c5c9e4cfdaa794a786b3d9164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:16.487099 18747 raft_consensus.cc:1261] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Refusing update from remote peer e8f9f23c5c9e4cfdaa794a786b3d9164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:16.487939 18837 consensus_queue.cc:1035] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:16.488143 18834 consensus_queue.cc:1035] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:16.497359 18842 mvcc.cc:205] Tried to move back new op lower bound from 7334473795522891776 to 7334473795364302848. Current Snapshot: MvccSnapshot[applied={T|T < 7334473795522891776}]
I20260929 00:50:16.501142 18841 mvcc.cc:205] Tried to move back new op lower bound from 7334473795522891776 to 7334473795364302848. Current Snapshot: MvccSnapshot[applied={T|T < 7334473795522891776}]
I20260929 00:50:16.509943 17547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:16.510144 17547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:16.514905 17547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f08ab8798c944c868d040a4f58fdee83: Sending DeleteTablet for 3 replicas of tablet 8adf8a2e7b0748689221cf837ac130dc
I20260929 00:50:16.515684 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adf8a2e7b0748689221cf837ac130dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:16 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:43970
I20260929 00:50:16.516299 18849 tablet_replica.cc:330] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: stopping tablet replica
I20260929 00:50:16.516849 18849 raft_consensus.cc:2229] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:16.517078 18727 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adf8a2e7b0748689221cf837ac130dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:16 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:55622
I20260929 00:50:16.517326 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adf8a2e7b0748689221cf837ac130dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:16 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:49630
I20260929 00:50:16.517560 18849 raft_consensus.cc:2258] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:16.517632 18850 tablet_replica.cc:330] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: stopping tablet replica
I20260929 00:50:16.517995 18850 raft_consensus.cc:2229] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:16.518316 18850 raft_consensus.cc:2258] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:16.519745 18850 ts_tablet_manager.cc:1905] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:16.519745 18849 ts_tablet_manager.cc:1905] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:16.521809 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 17516
I20260929 00:50:16.523738 18849 ts_tablet_manager.cc:1918] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:16.523749 18850 ts_tablet_manager.cc:1918] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:16.523869 18850 log.cc:1196] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/8adf8a2e7b0748689221cf837ac130dc
I20260929 00:50:16.523998 18849 log.cc:1196] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/8adf8a2e7b0748689221cf837ac130dc
I20260929 00:50:16.524232 18850 ts_tablet_manager.cc:1939] T 8adf8a2e7b0748689221cf837ac130dc P 47a69599c9e242d19b026d29b96d3e7b: Deleting consensus metadata
I20260929 00:50:16.524500 18849 ts_tablet_manager.cc:1939] T 8adf8a2e7b0748689221cf837ac130dc P e8f9f23c5c9e4cfdaa794a786b3d9164: Deleting consensus metadata
W20260929 00:50:16.530678 18410 connection.cc:535] server connection from 127.0.0.1:43970 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 00:50:16.530840 18681 connection.cc:535] server connection from 127.0.0.1:55622 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:50:16.532188 18851 tablet_replica.cc:330] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: stopping tablet replica
I20260929 00:50:16.532557 18851 raft_consensus.cc:2229] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:16.532948 18851 raft_consensus.cc:2258] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:16.534552 18851 ts_tablet_manager.cc:1905] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:16.535025 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.520388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.11.40.253:39543)
0929 00:50:16.520511 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.520515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.522793 (+ 2278us) server_negotiation.cc:407] Connection header received
0929 00:50:16.525660 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.525664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.525737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.525785 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.526677 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.527254 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.528122 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.528392 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.528517 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.530100 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.530104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.530107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.530183 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.532413 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.532987 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.532994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.532999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.533107 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.533975 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.533979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.533981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.534220 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.534341 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.534706 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.534862 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:50:16.537415 18851 ts_tablet_manager.cc:1918] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:16.537518 18851 log.cc:1196] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/8adf8a2e7b0748689221cf837ac130dc
I20260929 00:50:16.537832 18851 ts_tablet_manager.cc:1939] T 8adf8a2e7b0748689221cf837ac130dc P 312dcaf687e64010a9db6a66b73f068f: Deleting consensus metadata
W20260929 00:50:16.539106 18547 connection.cc:441] server connection from 127.0.0.1:49630 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49630 (request call id 1) could send its response
I20260929 00:50:16.546500 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.520169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.11.40.252:34399)
0929 00:50:16.520298 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.520303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.536509 (+ 16206us) server_negotiation.cc:407] Connection header received
0929 00:50:16.536704 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.536707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.536762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.536854 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.537746 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.538543 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.539694 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.540097 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.540207 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.542488 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.542492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.542495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.542526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.544397 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.545120 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.545124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.545126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.545214 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.545505 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.545508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.545510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.545722 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.545832 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.546181 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.546322 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:50:16.547284 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.534659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.11.40.253:39543)
0929 00:50:16.534802 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.534807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.535739 (+ 932us) server_negotiation.cc:407] Connection header received
0929 00:50:16.535906 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.535909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.535952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.535998 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.537422 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.538215 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.540146 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.540472 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.540549 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.542551 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.542555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.542557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.542583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.544550 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.545146 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.545149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.545151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.545235 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.545561 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.545565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.545566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.546124 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.546248 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.546968 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.547104 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:50:16.550688 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:16.540384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.11.40.252:34399)
0929 00:50:16.540499 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:16.540505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:16.541375 (+ 870us) server_negotiation.cc:407] Connection header received
0929 00:50:16.541600 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:16.541603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:16.541670 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:16.541740 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:16.542802 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.543600 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.544935 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:16.545273 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:16.545350 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:16.547093 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:16.547098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:16.547101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:16.547123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:16.548968 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.549426 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.549429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.549431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.549519 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:16.549806 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:16.549811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:16.549813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:16.549989 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:16.550133 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:16.550418 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 00:50:16.550533 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:50:17.504524 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:17.504043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:59961)
0929 00:50:17.504307 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:17.504395 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260929 00:50:17.505018 18658 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:40939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:17.505568 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:17.505058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:58349)
0929 00:50:17.505354 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:17.505462 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
W20260929 00:50:17.506271 18794 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:40939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:17.506980 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:17.506614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:57035)
0929 00:50:17.506831 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:17.506898 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
W20260929 00:50:17.507287 18521 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:40939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:18.015933 18863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f08ab8798c944c868d040a4f58fdee83)
I20260929 00:50:18.016086 18863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } }
I20260929 00:50:18.016680 18863 leader_election.cc:288] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939), f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253:39543)
I20260929 00:50:18.016934 18864 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f08ab8798c944c868d040a4f58fdee83)
I20260929 00:50:18.017132 18864 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } }
I20260929 00:50:18.017391 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.016852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50376)
0929 00:50:18.017197 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.017292 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
W20260929 00:50:18.017573 18325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:18.018276 18864 leader_election.cc:288] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939), 9442bd368dad459ab2090bda3ec8558e (127.11.40.252:34399)
I20260929 00:50:18.018628 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.018217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50388)
0929 00:50:18.018482 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.018537 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 00:50:18.018755 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.018377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50398)
0929 00:50:18.018602 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.018671 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
W20260929 00:50:18.018795 18325 leader_election.cc:334] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939): Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
W20260929 00:50:18.018900 18133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:18.019835 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.019586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50410)
0929 00:50:18.019710 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.019765 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 00:50:18.020020 18133 leader_election.cc:334] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939): Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:18.021778 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.017046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:39543 (local address 127.0.0.1:50280)
0929 00:50:18.017394 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.017408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:18.017585 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:18.018000 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:18.018003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:18.018023 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:18.018128 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.018135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.019220 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.019226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.020473 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.020482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.021367 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.021370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.021477 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.021498 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:50:18.021597 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:50:18.021677 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260929 00:50:18.021814 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.017039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.11.40.253:39543)
0929 00:50:18.017365 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.017370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.017561 (+ 191us) server_negotiation.cc:407] Connection header received
0929 00:50:18.017750 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.017753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.017828 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.017913 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:18.018285 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.019099 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.020596 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.021246 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.021358 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.021419 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.021624 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.021701 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
I20260929 00:50:18.022260 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9442bd368dad459ab2090bda3ec8558e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" is_pre_election: true
I20260929 00:50:18.022420 18159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9442bd368dad459ab2090bda3ec8558e in term 1.
I20260929 00:50:18.022795 18327 leader_election.cc:302] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9442bd368dad459ab2090bda3ec8558e, f7ef0b11ae2d4ed0b3a40220fd2c32a3; no voters: f08ab8798c944c868d040a4f58fdee83
I20260929 00:50:18.023080 18863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 00:50:18.023110 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.018678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:34399 (local address 127.0.0.1:39492)
0929 00:50:18.018870 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.018882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:18.018967 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:18.019743 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:18.019745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:18.019762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:18.019852 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.019858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.020752 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.020755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.021916 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.021928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.022706 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.022710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.022805 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.022827 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:50:18.022902 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 00:50:18.022974 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260929 00:50:18.023134 18863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f08ab8798c944c868d040a4f58fdee83)
I20260929 00:50:18.023176 18863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:50:18.023442 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.019268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.11.40.252:34399)
0929 00:50:18.019497 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.019508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.019523 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:50:18.019569 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.019572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.019619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.019732 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:18.020025 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.020642 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.022022 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.022579 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.022705 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.022760 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.023244 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.023317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260929 00:50:18.023838 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9442bd368dad459ab2090bda3ec8558e" is_pre_election: true
I20260929 00:50:18.024451 18863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } }
I20260929 00:50:18.024734 18351 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7ef0b11ae2d4ed0b3a40220fd2c32a3 in current term 2: Already voted for candidate 9442bd368dad459ab2090bda3ec8558e in this term.
I20260929 00:50:18.024863 18863 leader_election.cc:288] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 election: Requested vote from peers f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939), f7ef0b11ae2d4ed0b3a40220fd2c32a3 (127.11.40.253:39543)
I20260929 00:50:18.025250 18159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9442bd368dad459ab2090bda3ec8558e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3"
I20260929 00:50:18.025267 18133 leader_election.cc:302] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7ef0b11ae2d4ed0b3a40220fd2c32a3; no voters: 9442bd368dad459ab2090bda3ec8558e, f08ab8798c944c868d040a4f58fdee83
I20260929 00:50:18.025384 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.024979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50438)
0929 00:50:18.025261 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.025318 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260929 00:50:18.025398 18159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:50:18.026351 18159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9442bd368dad459ab2090bda3ec8558e in term 2.
I20260929 00:50:18.026495 18864 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:50:18.026859 18327 leader_election.cc:302] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9442bd368dad459ab2090bda3ec8558e, f7ef0b11ae2d4ed0b3a40220fd2c32a3; no voters:
I20260929 00:50:18.027062 18863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:50:18.027362 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.027039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50450)
0929 00:50:18.027193 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.027270 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 00:50:18.027345 18863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [term 2 LEADER]: Becoming Leader. State: Replica: 9442bd368dad459ab2090bda3ec8558e, State: Running, Role: LEADER
W20260929 00:50:18.027562 18325 leader_election.cc:334] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939): Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
I20260929 00:50:18.027690 18863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } }
I20260929 00:50:18.028872 18873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9442bd368dad459ab2090bda3ec8558e. Latest consensus state: current_term: 2 leader_uuid: "9442bd368dad459ab2090bda3ec8558e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:18.029006 18873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:18.029367 18875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:18.032024 18875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:18.032491 18875 catalog_manager.cc:1259] Loaded cluster ID: 71fae3a7ca6b4a8eb90f5b99d5adcccb
I20260929 00:50:18.032562 18875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:18.033227 18875 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:18.033612 18875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Loaded TSK: 0
I20260929 00:50:18.034291 18875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:18.089721 18341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:18.092972 18341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:18.096436 18159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [term 2 FOLLOWER]: Refusing update from remote peer 9442bd368dad459ab2090bda3ec8558e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 00:50:18.096505 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.096205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50470)
0929 00:50:18.096331 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.096403 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 00:50:18.097067 18873 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 00:50:18.097455 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.097210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.0.0.1:50482)
0929 00:50:18.097331 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.097392 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:50:18.097608 18325 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e -> Peer f08ab8798c944c868d040a4f58fdee83 (127.11.40.254:40939): Couldn't send request to peer f08ab8798c944c868d040a4f58fdee83. Status: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 00:50:18.098668 18864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9442bd368dad459ab2090bda3ec8558e. Latest consensus state: current_term: 2 leader_uuid: "9442bd368dad459ab2090bda3ec8558e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:18.098836 18864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:18.100083 18873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9442bd368dad459ab2090bda3ec8558e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:18.100236 18873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:18.100883 18864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9442bd368dad459ab2090bda3ec8558e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f08ab8798c944c868d040a4f58fdee83" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40939 } } peers { permanent_uuid: "f7ef0b11ae2d4ed0b3a40220fd2c32a3" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "9442bd368dad459ab2090bda3ec8558e" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 34399 } attrs { promote: false } } }
I20260929 00:50:18.101015 18864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ef0b11ae2d4ed0b3a40220fd2c32a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:50:18.112877 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.107445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35963 (local address 127.0.0.1:49640)
0929 00:50:18.107831 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.107851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:18.107984 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:18.108527 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:18.108530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:18.108553 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:18.108685 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.108694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.109683 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.109688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.111237 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.111247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.112380 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.112383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.112496 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.112521 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:50:18.112593 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:50:18.112683 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":65}
I20260929 00:50:18.113188 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.107647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38333 (local address 127.0.0.1:43994)
0929 00:50:18.108179 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.108197 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:18.108286 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:18.108967 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:18.108970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:18.108995 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:18.109073 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.109079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.110185 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.110190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.111550 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.111560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.112624 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.112628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.112719 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.112771 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 00:50:18.112962 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 00:50:18.113053 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":83}
I20260929 00:50:18.113391 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.107527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.11.40.194:35963)
0929 00:50:18.107841 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.107848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.107950 (+ 102us) server_negotiation.cc:407] Connection header received
0929 00:50:18.108107 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.108111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.108168 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.108488 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:18.108851 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.109554 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.111390 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.112012 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.112248 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.112323 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.112915 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.113214 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260929 00:50:18.113603 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.108042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:43241 (local address 127.0.0.1:55634)
0929 00:50:18.108706 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.108717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:18.108854 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:18.109882 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:18.109885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:18.109901 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:18.110045 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.110050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.110784 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.110787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.111837 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:18.111846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.113146 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.113149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:18.113248 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.113271 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:50:18.113351 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:50:18.113439 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":26,"thread_start_us":140,"threads_started":1}
I20260929 00:50:18.113915 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.107739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.11.40.193:38333)
0929 00:50:18.108621 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.108627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.108644 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:50:18.108701 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.108704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.108784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.108966 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:18.109199 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.110029 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.111688 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.112473 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.113561 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.113636 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.113679 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.113783 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":86,"threads_started":1}
I20260929 00:50:18.113984 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 647c05de43d74ea08ef3f0622c603d92 (DEFAULT_TABLE table=test-table [id=935b0208e61a4e7b9a239f19f36ed455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:18.114107 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.108174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.11.40.195:43241)
0929 00:50:18.108552 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.108556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.109678 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 00:50:18.109729 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.109731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.109782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.109873 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:18.110188 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.110671 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.111979 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.112758 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.113037 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.113099 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.113861 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.113971 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260929 00:50:18.114321 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647c05de43d74ea08ef3f0622c603d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:18.114400 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 647c05de43d74ea08ef3f0622c603d92 (DEFAULT_TABLE table=test-table [id=935b0208e61a4e7b9a239f19f36ed455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:18.114648 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647c05de43d74ea08ef3f0622c603d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:18.114590 18727 tablet_service.cc:1467] Processing CreateTablet for tablet 647c05de43d74ea08ef3f0622c603d92 (DEFAULT_TABLE table=test-table [id=935b0208e61a4e7b9a239f19f36ed455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:18.114858 18727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647c05de43d74ea08ef3f0622c603d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:18.116897 18885 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: Bootstrap starting.
I20260929 00:50:18.117928 18887 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Bootstrap starting.
I20260929 00:50:18.118255 18885 tablet_bootstrap.cc:654] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:18.119213 18887 tablet_bootstrap.cc:654] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:18.119580 18885 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: No bootstrap required, opened a new log
I20260929 00:50:18.119695 18885 ts_tablet_manager.cc:1397] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:18.120241 18885 raft_consensus.cc:348] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.120393 18885 raft_consensus.cc:374] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:18.120438 18885 raft_consensus.cc:729] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9f23c5c9e4cfdaa794a786b3d9164, State: Initialized, Role: FOLLOWER
I20260929 00:50:18.120326 18886 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Bootstrap starting.
I20260929 00:50:18.120594 18885 consensus_queue.cc:260] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.121340 18885 ts_tablet_manager.cc:1428] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:18.122860 18887 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: No bootstrap required, opened a new log
I20260929 00:50:18.122979 18887 ts_tablet_manager.cc:1397] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 00:50:18.123497 18887 raft_consensus.cc:348] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.123627 18887 raft_consensus.cc:374] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:18.123660 18887 raft_consensus.cc:729] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312dcaf687e64010a9db6a66b73f068f, State: Initialized, Role: FOLLOWER
I20260929 00:50:18.123808 18887 consensus_queue.cc:260] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.124151 18887 ts_tablet_manager.cc:1428] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:50:18.124584 18520 heartbeater.cc:503] Master 127.11.40.252:34399 was elected leader, sending a full tablet report...
I20260929 00:50:18.124735 18886 tablet_bootstrap.cc:654] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:18.125730 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.121965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:53777)
0929 00:50:18.125537 (+ 3572us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.125607 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3490,"thread_start_us":110,"threads_started":1}
I20260929 00:50:18.126415 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.124998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:58063)
0929 00:50:18.126128 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.126189 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":108,"threads_started":1}
I20260929 00:50:18.126724 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.126195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:33983)
0929 00:50:18.126601 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.126653 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":91,"threads_started":1}
I20260929 00:50:18.127081 18886 tablet_bootstrap.cc:492] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: No bootstrap required, opened a new log
I20260929 00:50:18.127107 18657 heartbeater.cc:503] Master 127.11.40.252:34399 was elected leader, sending a full tablet report...
I20260929 00:50:18.127216 18886 ts_tablet_manager.cc:1397] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 00:50:18.127769 18886 raft_consensus.cc:348] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.127913 18886 raft_consensus.cc:374] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:18.127964 18886 raft_consensus.cc:729] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a69599c9e242d19b026d29b96d3e7b, State: Initialized, Role: FOLLOWER
I20260929 00:50:18.128125 18886 consensus_queue.cc:260] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.128334 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.127785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:51913)
0929 00:50:18.128138 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.128197 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 00:50:18.128484 18886 ts_tablet_manager.cc:1428] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:50:18.129465 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.128991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:56875)
0929 00:50:18.129288 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.129344 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
I20260929 00:50:18.130378 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.130127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:57897)
0929 00:50:18.130252 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.130310 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:50:18.130378 18793 heartbeater.cc:503] Master 127.11.40.252:34399 was elected leader, sending a full tablet report...
I20260929 00:50:18.292537 18890 raft_consensus.cc:482] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:18.292752 18890 raft_consensus.cc:504] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.293301 18890 leader_election.cc:288] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963), 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:18.293850 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647c05de43d74ea08ef3f0622c603d92" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312dcaf687e64010a9db6a66b73f068f" is_pre_election: true
I20260929 00:50:18.293942 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647c05de43d74ea08ef3f0622c603d92" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a69599c9e242d19b026d29b96d3e7b" is_pre_election: true
I20260929 00:50:18.294051 18611 raft_consensus.cc:2454] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 0.
I20260929 00:50:18.294107 18747 raft_consensus.cc:2454] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 0.
I20260929 00:50:18.294489 18407 leader_election.cc:302] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312dcaf687e64010a9db6a66b73f068f, e8f9f23c5c9e4cfdaa794a786b3d9164; no voters:
I20260929 00:50:18.294672 18890 raft_consensus.cc:2781] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:18.294739 18890 raft_consensus.cc:482] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:18.294770 18890 raft_consensus.cc:3037] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:18.295763 18890 raft_consensus.cc:504] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.296118 18890 leader_election.cc:288] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 election: Requested vote from peers 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963), 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241)
I20260929 00:50:18.296506 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647c05de43d74ea08ef3f0622c603d92" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312dcaf687e64010a9db6a66b73f068f"
I20260929 00:50:18.296547 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647c05de43d74ea08ef3f0622c603d92" candidate_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a69599c9e242d19b026d29b96d3e7b"
I20260929 00:50:18.296620 18611 raft_consensus.cc:3037] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:18.296691 18747 raft_consensus.cc:3037] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:18.297619 18611 raft_consensus.cc:2454] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 1.
I20260929 00:50:18.297695 18747 raft_consensus.cc:2454] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f9f23c5c9e4cfdaa794a786b3d9164 in term 1.
I20260929 00:50:18.297942 18407 leader_election.cc:302] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312dcaf687e64010a9db6a66b73f068f, e8f9f23c5c9e4cfdaa794a786b3d9164; no voters:
I20260929 00:50:18.298162 18890 raft_consensus.cc:2781] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:18.298372 18890 raft_consensus.cc:686] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 LEADER]: Becoming Leader. State: Replica: e8f9f23c5c9e4cfdaa794a786b3d9164, State: Running, Role: LEADER
I20260929 00:50:18.298524 18890 consensus_queue.cc:237] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } }
I20260929 00:50:18.299227 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.298917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:39511)
0929 00:50:18.299077 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.299136 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 00:50:18.300355 18340 catalog_manager.cc:5495] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f9f23c5c9e4cfdaa794a786b3d9164" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 } health_report { overall_health: UNKNOWN } } }
W20260929 00:50:18.302848 18796 tablet.cc:2369] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:18.320550 18660 tablet.cc:2369] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:18.325977 18523 tablet.cc:2369] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:18.357231 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.353884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.11.40.193:38333)
0929 00:50:18.354063 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:18.354067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:18.354085 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:50:18.354143 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:18.354145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:18.354188 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:18.354273 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:18.354563 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.355154 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.356064 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:18.356294 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:18.356414 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:18.356676 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:18.356758 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:18.356893 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:18.357018 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 00:50:18.357077 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:50:18.359731 18747 raft_consensus.cc:1261] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Refusing update from remote peer e8f9f23c5c9e4cfdaa794a786b3d9164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:18.359738 18611 raft_consensus.cc:1261] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Refusing update from remote peer e8f9f23c5c9e4cfdaa794a786b3d9164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:18.360296 18890 consensus_queue.cc:1035] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "312dcaf687e64010a9db6a66b73f068f" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:18.360505 18898 consensus_queue.cc:1035] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a69599c9e242d19b026d29b96d3e7b" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 43241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:18.362115 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.361844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:56537)
0929 00:50:18.361951 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.362020 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 00:50:18.363773 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.363303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:32973)
0929 00:50:18.363630 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.363688 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260929 00:50:18.367914 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.367600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.193:54227)
0929 00:50:18.367729 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.367832 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:50:18.368366 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.367852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.195:41291)
0929 00:50:18.368222 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.368292 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260929 00:50:18.368405 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.368103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:40939 (local address 127.11.40.194:34171)
0929 00:50:18.368289 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.368335 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 00:50:18.370414 18339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:18.370656 18339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:18.374724 18339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9442bd368dad459ab2090bda3ec8558e: Sending DeleteTablet for 3 replicas of tablet 647c05de43d74ea08ef3f0622c603d92
I20260929 00:50:18.375438 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 647c05de43d74ea08ef3f0622c603d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:18 UTC) from {username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:49640
I20260929 00:50:18.375671 18727 tablet_service.cc:1514] Processing DeleteTablet for tablet 647c05de43d74ea08ef3f0622c603d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:18 UTC) from {username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:55634
I20260929 00:50:18.375886 18910 tablet_replica.cc:330] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: stopping tablet replica
I20260929 00:50:18.376006 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 647c05de43d74ea08ef3f0622c603d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:18 UTC) from {username='oryx', principal='oryx/127.11.40.252@KRBTEST.COM'} at 127.0.0.1:43994
I20260929 00:50:18.376047 18910 raft_consensus.cc:2229] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:18.376052 18911 tablet_replica.cc:330] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: stopping tablet replica
I20260929 00:50:18.376160 18910 raft_consensus.cc:2258] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:18.376176 18911 raft_consensus.cc:2229] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:18.376313 18911 raft_consensus.cc:2258] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:18.376911 18910 ts_tablet_manager.cc:1905] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:18.377012 18912 tablet_replica.cc:330] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164: stopping tablet replica
I20260929 00:50:18.377058 18911 ts_tablet_manager.cc:1905] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:18.377118 18912 raft_consensus.cc:2229] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:18.377393 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18293
I20260929 00:50:18.377425 18912 raft_consensus.cc:2258] T 647c05de43d74ea08ef3f0622c603d92 P e8f9f23c5c9e4cfdaa794a786b3d9164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:18.379710 18910 ts_tablet_manager.cc:1918] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:18.379770 18910 log.cc:1196] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/647c05de43d74ea08ef3f0622c603d92
I20260929 00:50:18.380007 18910 ts_tablet_manager.cc:1939] T 647c05de43d74ea08ef3f0622c603d92 P 312dcaf687e64010a9db6a66b73f068f: Deleting consensus metadata
I20260929 00:50:18.380432 18911 ts_tablet_manager.cc:1918] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:18.380512 18911 log.cc:1196] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/647c05de43d74ea08ef3f0622c603d92
I20260929 00:50:18.380954 18911 ts_tablet_manager.cc:1939] T 647c05de43d74ea08ef3f0622c603d92 P 47a69599c9e242d19b026d29b96d3e7b: Deleting consensus metadata
I20260929 00:50:18.382252 18325 catalog_manager.cc:4841] TS 312dcaf687e64010a9db6a66b73f068f (127.11.40.194:35963): tablet 647c05de43d74ea08ef3f0622c603d92 (table test-table [id=935b0208e61a4e7b9a239f19f36ed455]) successfully deleted
I20260929 00:50:18.382504 18328 catalog_manager.cc:4841] TS 47a69599c9e242d19b026d29b96d3e7b (127.11.40.195:43241): tablet 647c05de43d74ea08ef3f0622c603d92 (table test-table [id=935b0208e61a4e7b9a239f19f36ed455]) successfully deleted
I20260929 00:50:18.388001 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18528
I20260929 00:50:18.388911 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:18.388185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:38333 (local address 127.0.0.1:44010)
0929 00:50:18.388698 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:18.388768 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20260929 00:50:18.389853 18326 catalog_manager.cc:4584] TS e8f9f23c5c9e4cfdaa794a786b3d9164 (127.11.40.193:38333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 647c05de43d74ea08ef3f0622c603d92: Network error: Client connection negotiation failed: client connection to 127.11.40.193:38333: connect: Connection refused (error 111)
I20260929 00:50:18.394309 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18664
I20260929 00:50:18.400568 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18012
I20260929 00:50:18.406939 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18121
2026-09-29T00:50:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5773 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:18 dist-test-slave-5309 krb5kdc[18915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:18 dist-test-slave-5309 krb5kdc[18915](info): set up 2 sockets
Sep 29 00:50:18 dist-test-slave-5309 krb5kdc[18915](info): commencing operation
krb5kdc: starting...
W20260929 00:50:19.471755 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:19 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab.
Entry for principal oryx/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab.
I20260929 00:50:19.613415 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42451
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab
--principal=oryx/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:19.728768 18928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:19.729214 18928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:19.729264 18928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:19.733546 18928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:19.733621 18928 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:50:19.733639 18928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:19.733657 18928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:19.733675 18928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:19.733690 18928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:19.739603 18928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:42451
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.254.keytab
--principal=oryx/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:19.740869 18928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:19.741961 18928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:19.750785 18936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:19.750885 18928 server_base.cc:1034] running on GCE node
W20260929 00:50:19.750795 18934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:19.750814 18933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:19.751904 18928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:19.753181 18928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:19.754377 18928 hybrid_clock.cc:648] HybridClock initialized: now 1790643019754349 us; error 43 us; skew 500 ppm
Sep 29 00:50:19 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643019, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:19.757387 18928 init.cc:377] Logged in from keytab as oryx/127.11.40.254@KRBTEST.COM (short username oryx)
I20260929 00:50:19.758354 18928 webserver.cc:460] Webserver started at http://127.11.40.254:34421/ using document root <none> and password file <none>
I20260929 00:50:19.758926 18928 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:19.758988 18928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:19.759363 18928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:19.761731 18928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "72c1bd56abd14ed484918a1317f9e180"
format_stamp: "Formatted at 2026-09-29 00:50:19 on dist-test-slave-5309"
server_key: "0771fecd7f8d3e090b8586cea9b6e76f"
server_key_iv: "b45ab5128238c6657ed40bbd7c7fe174"
server_key_version: "encryptionkey@0"
I20260929 00:50:19.762356 18928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "72c1bd56abd14ed484918a1317f9e180"
format_stamp: "Formatted at 2026-09-29 00:50:19 on dist-test-slave-5309"
server_key: "0771fecd7f8d3e090b8586cea9b6e76f"
server_key_iv: "b45ab5128238c6657ed40bbd7c7fe174"
server_key_version: "encryptionkey@0"
I20260929 00:50:19.766602 18928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 00:50:19.769507 18943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:19.770890 18928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:19.771039 18928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "72c1bd56abd14ed484918a1317f9e180"
format_stamp: "Formatted at 2026-09-29 00:50:19 on dist-test-slave-5309"
server_key: "0771fecd7f8d3e090b8586cea9b6e76f"
server_key_iv: "b45ab5128238c6657ed40bbd7c7fe174"
server_key_version: "encryptionkey@0"
I20260929 00:50:19.771163 18928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:19.791846 18928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:19.796620 18928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:19.796846 18928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:19.806842 18928 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:42451
I20260929 00:50:19.806864 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:42451 every 8 connection(s)
I20260929 00:50:19.807863 18928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:19.810776 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:19.810918 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 18928
I20260929 00:50:19.811075 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:19.811404 11427 external_mini_cluster.cc:1428] Setting key 2d5bd4e755a7142321aface4839ccd45
I20260929 00:50:19.818137 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Bootstrap starting.
I20260929 00:50:19.820647 19006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Neither blocks nor log segments found. Creating new log.
Sep 29 00:50:19 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:19.821488 19006 log.cc:824] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:19.824093 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: No bootstrap required, opened a new log
I20260929 00:50:19.827453 19006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } }
I20260929 00:50:19.827634 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:19.812985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.11.40.254:42451)
0929 00:50:19.813431 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:19.813460 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:19.813506 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:50:19.814290 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:19.814314 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:19.814764 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:19.815141 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:19.815812 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:19.817358 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:19.818228 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:19.818914 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:19.819073 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:19.821883 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:19.821891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:19.821909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:19.821941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:19.824626 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:19.825160 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:19.825166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:19.825171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:19.825286 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:19.825589 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:19.825592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:19.825594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:19.826010 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:19.826223 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:19.826522 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 00:50:19.826824 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260929 00:50:19.827682 19006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:19.827787 19006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c1bd56abd14ed484918a1317f9e180, State: Initialized, Role: FOLLOWER
I20260929 00:50:19.828260 19006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } }
I20260929 00:50:19.828387 19006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:19.828473 19006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:19.828575 19006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:19.829599 19006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } }
I20260929 00:50:19.829988 19006 leader_election.cc:302] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72c1bd56abd14ed484918a1317f9e180; no voters:
I20260929 00:50:19.830341 19006 leader_election.cc:288] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:19.830533 19011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:19.830782 19011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [term 1 LEADER]: Becoming Leader. State: Replica: 72c1bd56abd14ed484918a1317f9e180, State: Running, Role: LEADER
I20260929 00:50:19.831117 19011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } }
I20260929 00:50:19.831739 19006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:19.833101 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72c1bd56abd14ed484918a1317f9e180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } } }
I20260929 00:50:19.833240 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:19.833083 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72c1bd56abd14ed484918a1317f9e180. Latest consensus state: current_term: 1 leader_uuid: "72c1bd56abd14ed484918a1317f9e180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72c1bd56abd14ed484918a1317f9e180" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42451 } } }
I20260929 00:50:19.833496 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180 [sys.catalog]: This master's current role is: LEADER
W20260929 00:50:19.836602 19026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:50:19.836696 19026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:50:19.836771 19021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:19.837541 19021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:19.843420 19021 catalog_manager.cc:1347] Generated new cluster ID: 0cf111b697d14556866f22106bec134c
I20260929 00:50:19.843503 19021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:19.859988 19021 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:19.861371 19021 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:19.870386 19021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Generated new TSK 0
I20260929 00:50:19.871322 19021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
I20260929 00:50:19.909235 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42451
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:20.028718 19032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:20.029055 19032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:20.029102 19032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:20.033589 19032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:20.033667 19032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:20.033710 19032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:50:20.033744 19032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:20.039425 19032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:20.040532 19032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:20.041688 19032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:20.052793 19040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:20.053000 19037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:20.053037 19038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:20.052995 19032 server_base.cc:1034] running on GCE node
I20260929 00:50:20.053632 19032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:20.054311 19032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:20.055548 19032 hybrid_clock.cc:648] HybridClock initialized: now 1790643020055510 us; error 50 us; skew 500 ppm
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:20.058627 19032 init.cc:377] Logged in from keytab as oryx/127.11.40.193@KRBTEST.COM (short username oryx)
I20260929 00:50:20.059561 19032 webserver.cc:460] Webserver started at http://127.11.40.193:32991/ using document root <none> and password file <none>
I20260929 00:50:20.060241 19032 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:20.060353 19032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:20.060596 19032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:20.062803 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "74c9fe5c1d53e86361d4a35d95c2c318"
server_key_iv: "ee5adc8a641356fb0da3d01554d74151"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.063397 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "74c9fe5c1d53e86361d4a35d95c2c318"
server_key_iv: "ee5adc8a641356fb0da3d01554d74151"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.067396 19032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:20.070024 19047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.071367 19032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:20.071509 19032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "74c9fe5c1d53e86361d4a35d95c2c318"
server_key_iv: "ee5adc8a641356fb0da3d01554d74151"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.071630 19032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:20.082872 19032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:20.086324 19032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:20.086552 19032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:20.088025 19032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:20.088096 19032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.088164 19032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:20.088212 19032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:20.099220 19032 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43011
I20260929 00:50:20.099267 19162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43011 every 8 connection(s)
I20260929 00:50:20.100149 19032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:20.103024 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.089048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:39909 (local address 127.11.40.254:42451)
0929 00:50:20.089180 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.089184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.090281 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 00:50:20.091527 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.091534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.091608 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.091712 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.093604 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.094531 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.095775 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.096036 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.096180 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.099214 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.099218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.099221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.099259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.101134 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.101637 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.101640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.101641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.101721 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.102059 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.102062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.102063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.102242 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.102314 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.102774 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.102890 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:50:20.103928 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.089615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.193:39909)
0929 00:50:20.090116 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.090160 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.091217 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.092072 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.092082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.092590 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.093399 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.093422 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.094661 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.094665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.095643 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.095653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.096231 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.096235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.096409 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.097082 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.097107 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.099017 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.101249 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.101254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.101265 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.101533 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.101798 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.101800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.101801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.101957 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.102327 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.102332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.102562 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.102709 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.102990 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260929 00:50:20.105278 19163 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42451
I20260929 00:50:20.105446 19163 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:20.105959 19163 heartbeater.cc:511] Master 127.11.40.254:42451 requested a full tablet report, sending...
I20260929 00:50:20.106557 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19032
I20260929 00:50:20.106703 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:20.107033 11427 external_mini_cluster.cc:1428] Setting key 5ee3d4763779c2494bfe8977bfe8e932
I20260929 00:50:20.107707 18960 ts_manager.cc:194] Registered new tserver with Master: 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011)
I20260929 00:50:20.109285 18960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:39909
I20260929 00:50:20.116722 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.107766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:33951 (local address 127.11.40.254:42451)
0929 00:50:20.107872 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.107876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.108120 (+ 244us) server_negotiation.cc:407] Connection header received
0929 00:50:20.108355 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.108359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.108407 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.108485 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.109095 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.109788 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.110985 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.111367 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.111500 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.113073 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.113082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.113085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.113112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.114736 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.115351 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.115354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.115356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.115444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.115832 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.115835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.115836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.116035 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.116117 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.116466 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.116586 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:50:20.116921 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.107609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.193:33951)
0929 00:50:20.108019 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.108034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.108135 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.108588 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.108591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.108831 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.108958 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.108965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.109997 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.110000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.110845 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.110856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.111515 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.111518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.111614 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.112178 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.112196 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.112770 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.114883 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.114889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.114895 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.115211 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.115552 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.115558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.115561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.115675 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.116148 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.116153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.116279 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.116400 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.116604 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.11.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
I20260929 00:50:20.141702 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42451
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:20.264496 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:20.264943 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:20.265033 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:20.270004 19172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:20.270077 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:20.270120 19172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:50:20.270154 19172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:20.275959 19172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:20.277220 19172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:20.278324 19172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:20.288746 19178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:20.288841 19172 server_base.cc:1034] running on GCE node
W20260929 00:50:20.288755 19177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:20.288868 19180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:20.290174 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:20.290854 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:20.292104 19172 hybrid_clock.cc:648] HybridClock initialized: now 1790643020292052 us; error 71 us; skew 500 ppm
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:20.295463 19172 init.cc:377] Logged in from keytab as oryx/127.11.40.194@KRBTEST.COM (short username oryx)
I20260929 00:50:20.296578 19172 webserver.cc:460] Webserver started at http://127.11.40.194:42429/ using document root <none> and password file <none>
I20260929 00:50:20.297281 19172 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:20.297384 19172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:20.297633 19172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:20.300010 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "48b0203088bf47a1a064708429e117bd"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "ae1ab7cfc1591ba53ceee6b465daeb58"
server_key_iv: "2254ce67e17cf57af060c16ba5cb977c"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.300637 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "48b0203088bf47a1a064708429e117bd"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "ae1ab7cfc1591ba53ceee6b465daeb58"
server_key_iv: "2254ce67e17cf57af060c16ba5cb977c"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.304793 19172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:50:20.308010 19187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.309829 19172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:20.310029 19172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "48b0203088bf47a1a064708429e117bd"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "ae1ab7cfc1591ba53ceee6b465daeb58"
server_key_iv: "2254ce67e17cf57af060c16ba5cb977c"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.310185 19172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:20.331833 19172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:20.335402 19172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:20.335568 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:20.337040 19172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:20.337121 19172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.337193 19172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:20.337245 19172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.348059 19172 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:37153
I20260929 00:50:20.348133 19302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:37153 every 8 connection(s)
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:20.349233 19172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:20.349679 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19172
I20260929 00:50:20.349807 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:20.350136 11427 external_mini_cluster.cc:1428] Setting key 84309de5eb73318f16c4cc9e4ff0c172
I20260929 00:50:20.353917 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.337953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56613 (local address 127.11.40.254:42451)
0929 00:50:20.338077 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.338081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.339207 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 00:50:20.340508 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.340515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.340585 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.340690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.342386 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.343325 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.345158 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.345572 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.345710 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.349240 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.349248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.349253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.349290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.351100 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.351851 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.351854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.351855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.351952 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.352446 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.352449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.352451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.352649 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.352733 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.353490 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.353654 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:50:20.354777 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.338536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.194:56613)
0929 00:50:20.339050 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.339096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.340256 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.340948 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.340958 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.341434 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.342209 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.342230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.343496 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.343499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.345015 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.345027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.345837 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.345843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.346071 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.346990 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.347013 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.348995 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.351225 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.351231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.351308 (+ 77us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.351705 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.352060 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.352066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.352069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.352279 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.352760 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.352771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.353143 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.353370 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.353766 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":151,"threads_started":1}
I20260929 00:50:20.356208 19303 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42451
I20260929 00:50:20.356355 19303 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:20.357010 19303 heartbeater.cc:511] Master 127.11.40.254:42451 requested a full tablet report, sending...
I20260929 00:50:20.358269 18954 ts_manager.cc:194] Registered new tserver with Master: 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:20.359140 18954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:56613
WARNING: no policy specified for oryx/127.11.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.195@KRBTEST.COM" created.
I20260929 00:50:20.366782 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.358570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.194:44295)
0929 00:50:20.358893 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.358904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.359020 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.359286 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.359289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.359519 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.359648 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.359655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.360968 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.360971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.361625 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.361634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.362131 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.362134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.362205 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.362773 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.362786 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.363173 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.365118 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.365123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.365126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.365345 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.365636 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.365641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.365643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.365736 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.366163 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.366167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.366273 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.366390 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.366562 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260929 00:50:20.366775 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.358626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:44295 (local address 127.11.40.254:42451)
0929 00:50:20.358759 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.358763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.358979 (+ 216us) server_negotiation.cc:407] Connection header received
0929 00:50:20.359131 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.359134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.359182 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.359265 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.359781 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.360690 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.361743 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.362024 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.362110 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.363323 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.363328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.363330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.363361 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.365004 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.365449 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.365454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.365456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.365541 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.365833 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.365837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.365838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.366061 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.366175 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.366449 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.366574 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
I20260929 00:50:20.382094 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42451
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:20.502372 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:20.502664 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:20.502753 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:20.507125 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:20.507205 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:20.507247 19312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:50:20.507316 19312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:20.512976 19312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.40.254:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:20.514189 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:20.515362 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:20.525537 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:20.525614 19312 server_base.cc:1034] running on GCE node
W20260929 00:50:20.525537 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:20.525544 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:20.526307 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:20.526983 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:20.528196 19312 hybrid_clock.cc:648] HybridClock initialized: now 1790643020528180 us; error 26 us; skew 500 ppm
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:20.531162 19312 init.cc:377] Logged in from keytab as oryx/127.11.40.195@KRBTEST.COM (short username oryx)
I20260929 00:50:20.532191 19312 webserver.cc:460] Webserver started at http://127.11.40.195:34303/ using document root <none> and password file <none>
I20260929 00:50:20.532960 19312 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:20.533070 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:20.533300 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:20.535411 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "7d79581e63ba4fe3900d89b44d061f90"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "9cceba7ea4ede27b25dbb5bb8ce60222"
server_key_iv: "6b6c461698c5ef804ea13faf35898086"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.536015 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "7d79581e63ba4fe3900d89b44d061f90"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "9cceba7ea4ede27b25dbb5bb8ce60222"
server_key_iv: "6b6c461698c5ef804ea13faf35898086"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.540086 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 00:50:20.543146 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.545455 19312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 00:50:20.545631 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "7d79581e63ba4fe3900d89b44d061f90"
format_stamp: "Formatted at 2026-09-29 00:50:20 on dist-test-slave-5309"
server_key: "9cceba7ea4ede27b25dbb5bb8ce60222"
server_key_iv: "6b6c461698c5ef804ea13faf35898086"
server_key_version: "encryptionkey@0"
I20260929 00:50:20.545794 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:20.562284 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:20.565853 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:20.566087 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:20.567651 19312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:20.567726 19312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:20.567794 19312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:20.567836 19312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:20.579097 19312 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:41507
I20260929 00:50:20.579160 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:41507 every 8 connection(s)
I20260929 00:50:20.580065 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:20.582932 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.568550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:57545 (local address 127.11.40.254:42451)
0929 00:50:20.568713 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.568717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.569820 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 00:50:20.571208 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.571216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.571293 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.571357 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.573453 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.574066 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.575833 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.576198 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.576325 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.579031 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.579037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.579041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.579074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.580746 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.581399 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.581402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.581404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.581488 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.581869 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.581873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.581874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.582086 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.582180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.582689 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.582807 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:50:20.583793 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.569099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.195:57545)
0929 00:50:20.569613 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.569679 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.570876 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.571849 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.571860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.572417 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.573259 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.573286 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.574264 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.574267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.575666 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.575684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.576377 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.576381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.576571 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.577264 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.577296 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.578845 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.580967 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.580973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.580987 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.581288 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.581570 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.581572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.581573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.581701 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.582189 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.582194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.582418 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.582568 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.582820 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260929 00:50:20.585196 19443 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42451
I20260929 00:50:20.585383 19443 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:20.585887 19443 heartbeater.cc:511] Master 127.11.40.254:42451 requested a full tablet report, sending...
I20260929 00:50:20.587016 18960 ts_manager.cc:194] Registered new tserver with Master: 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:20.587615 18960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:57545
I20260929 00:50:20.589562 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19312
I20260929 00:50:20.589726 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:20.590025 11427 external_mini_cluster.cc:1428] Setting key b6e490548ec7c8510ff19f91a6cc2808
I20260929 00:50:20.592463 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:50:20.595634 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.587674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.11.40.195:38797)
0929 00:50:20.587996 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.588009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.588132 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.588373 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.588376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.588588 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:20.588683 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.588688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.589576 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.589580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.590272 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.590281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.590894 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.590900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.591006 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.591624 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:20.591639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:20.591992 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:20.593955 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.593961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.593966 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.594259 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.594635 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:20.594637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:20.594639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:20.594716 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:20.595173 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:20.595175 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:20.595262 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.595348 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.595490 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":117,"threads_started":1}
I20260929 00:50:20.595813 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.587744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:38797 (local address 127.11.40.254:42451)
0929 00:50:20.587870 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.587875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.588089 (+ 214us) server_negotiation.cc:407] Connection header received
0929 00:50:20.588223 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.588226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.588271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.588357 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.588796 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.589414 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.590405 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.590723 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.590850 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.592123 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.592130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.592133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.592159 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.593834 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.594425 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.594432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.594435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.594538 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.594804 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.594810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.594812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.595071 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.595183 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.595446 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.595581 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:50:20.619589 19009 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:20.612337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.11.40.254:42451)
0929 00:50:20.612506 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.612514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.612530 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:20.612580 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.612582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.612627 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.612744 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.613723 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.614441 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.615470 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.615717 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.615815 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.619431 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40654: BlockingRecv error: recv got EOF from 127.0.0.1:40654 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 00:50:20 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:20.629668 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.621024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.11.40.254:42451)
0929 00:50:20.621145 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.621151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.621317 (+ 166us) server_negotiation.cc:407] Connection header received
0929 00:50:20.621515 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.621519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.621569 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.621659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:20.622271 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.622786 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.623614 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.623833 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.623949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.626361 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:20.626365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:20.626367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:20.626391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:20.627901 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.628390 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.628394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.628396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.628485 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:20.628760 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:20.628764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:20.628766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:20.629009 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:20.629155 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.629388 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.629502 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:50:20.632694 18960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:20.635876 18960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:20.650869 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.644744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.11.40.194:37153)
0929 00:50:20.645203 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.645217 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.645241 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:50:20.645947 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.645952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.646175 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.646308 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.646862 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.648022 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.649489 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650126 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.650270 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.650467 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.650602 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.650697 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":144,"threads_started":1}
I20260929 00:50:20.650921 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.645307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.0.0.1:43758)
0929 00:50:20.645575 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.645589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.645678 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.646529 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.646532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.646545 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.646654 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.646661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.647687 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.647693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.649365 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.649376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650558 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.650562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.650640 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.650669 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.650710 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.650759 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":47,"thread_start_us":78,"threads_started":1}
I20260929 00:50:20.651005 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.644701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43011 (local address 127.0.0.1:60968)
0929 00:50:20.645384 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.645398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.645527 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.646389 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.646395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.646421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.646589 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.646595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.648130 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.648134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.649404 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.649414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650497 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.650502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.650599 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.650629 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.650676 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.650754 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":69,"thread_start_us":151,"threads_started":1}
I20260929 00:50:20.651304 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.644561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:37153 (local address 127.0.0.1:37238)
0929 00:50:20.645109 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.645161 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.645772 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.646516 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.646519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.646537 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.646663 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.646678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.648157 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.648160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.649365 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.649374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650251 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.650255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.650364 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.650402 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.650566 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.650697 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":62,"thread_start_us":67,"threads_started":1}
I20260929 00:50:20.651376 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.645366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.11.40.193:43011)
0929 00:50:20.645909 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.645915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.645948 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:50:20.646033 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.646037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.646220 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.646750 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.646925 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.648003 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.649532 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650350 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.650842 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.651084 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.651140 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.651251 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":99,"threads_started":1}
I20260929 00:50:20.651844 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.645482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.11.40.195:41507)
0929 00:50:20.645824 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.645839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.645861 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:50:20.645944 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.645950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.646352 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.646539 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.646788 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.647568 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.649522 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.650350 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.651262 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.651562 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.651620 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.651713 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260929 00:50:20.654217 19237 tablet_service.cc:1467] Processing CreateTablet for tablet 842ae25300854c21b55d97c454fe3b98 (DEFAULT_TABLE table=test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.654532 19377 tablet_service.cc:1467] Processing CreateTablet for tablet 842ae25300854c21b55d97c454fe3b98 (DEFAULT_TABLE table=test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.654525 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 842ae25300854c21b55d97c454fe3b98 (DEFAULT_TABLE table=test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.655316 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842ae25300854c21b55d97c454fe3b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.655560 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842ae25300854c21b55d97c454fe3b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.655537 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842ae25300854c21b55d97c454fe3b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.662676 19471 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Bootstrap starting.
I20260929 00:50:20.663708 19472 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Bootstrap starting.
I20260929 00:50:20.663980 19473 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Bootstrap starting.
I20260929 00:50:20.665360 19471 tablet_bootstrap.cc:654] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.665714 19473 tablet_bootstrap.cc:654] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.665778 19472 tablet_bootstrap.cc:654] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.666299 19471 log.cc:824] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:20.666488 19473 log.cc:824] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:20.666618 19472 log.cc:824] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:20.668504 19473 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: No bootstrap required, opened a new log
I20260929 00:50:20.668587 19472 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: No bootstrap required, opened a new log
I20260929 00:50:20.668587 19471 tablet_bootstrap.cc:492] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: No bootstrap required, opened a new log
I20260929 00:50:20.668726 19473 ts_tablet_manager.cc:1397] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 00:50:20.668766 19471 ts_tablet_manager.cc:1397] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:50:20.668766 19472 ts_tablet_manager.cc:1397] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:50:20.671603 19473 raft_consensus.cc:348] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.671860 19473 raft_consensus.cc:374] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.671943 19473 raft_consensus.cc:729] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.672420 19473 consensus_queue.cc:260] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.672794 19471 raft_consensus.cc:348] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.673106 19443 heartbeater.cc:503] Master 127.11.40.254:42451 was elected leader, sending a full tablet report...
I20260929 00:50:20.673177 19471 raft_consensus.cc:374] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.673233 19471 raft_consensus.cc:729] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.673239 19472 raft_consensus.cc:348] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.673532 19472 raft_consensus.cc:374] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.673580 19472 raft_consensus.cc:729] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.673683 19471 consensus_queue.cc:260] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.673951 19472 consensus_queue.cc:260] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.674572 19473 ts_tablet_manager.cc:1428] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 00:50:20.674796 19472 ts_tablet_manager.cc:1428] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:50:20.674950 19471 ts_tablet_manager.cc:1428] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 00:50:20.675364 19303 heartbeater.cc:503] Master 127.11.40.254:42451 was elected leader, sending a full tablet report...
I20260929 00:50:20.675388 19163 heartbeater.cc:503] Master 127.11.40.254:42451 was elected leader, sending a full tablet report...
I20260929 00:50:20.785144 19478 raft_consensus.cc:482] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:20.785413 19478 raft_consensus.cc:504] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.786695 19478 leader_election.cc:288] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:20.791761 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.787052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.11.40.194:58615)
0929 00:50:20.787181 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.787198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.787305 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.787638 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.787640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.787661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.787770 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.787775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.788980 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.788985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.790314 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.790325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.791307 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.791310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.791411 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.791432 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.791522 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.791603 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 00:50:20.791833 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.787125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:58615 (local address 127.11.40.195:41507)
0929 00:50:20.787259 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.787264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.787280 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:20.787453 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.787457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.787520 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.787608 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.787931 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.788750 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.790448 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.791075 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.791198 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.791250 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.791547 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.791670 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:50:20.793054 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.787059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43011 (local address 127.11.40.194:58935)
0929 00:50:20.787882 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.787899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.788037 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.788438 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.788441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.788458 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.789305 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.789314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.790447 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.790450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.791546 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.791554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.792594 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.792599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.792699 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.792722 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.792816 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.792888 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":365,"thread_start_us":158,"threads_started":1}
I20260929 00:50:20.793035 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842ae25300854c21b55d97c454fe3b98" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90" is_pre_election: true
I20260929 00:50:20.793449 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.787149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:58935 (local address 127.11.40.193:43011)
0929 00:50:20.787290 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.787293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.787998 (+ 705us) server_negotiation.cc:407] Connection header received
0929 00:50:20.788236 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.788239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.788311 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.788393 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.789536 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.790296 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.791673 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.792313 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.792432 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.792485 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.792865 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.792980 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:50:20.793422 19397 raft_consensus.cc:2454] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 0.
I20260929 00:50:20.794167 19189 leader_election.cc:302] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 7d79581e63ba4fe3900d89b44d061f90; no voters:
I20260929 00:50:20.794595 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842ae25300854c21b55d97c454fe3b98" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" is_pre_election: true
I20260929 00:50:20.794898 19117 raft_consensus.cc:2454] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 0.
I20260929 00:50:20.796962 19478 raft_consensus.cc:2781] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:20.797116 19478 raft_consensus.cc:482] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:20.797210 19478 raft_consensus.cc:3037] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.798358 19478 raft_consensus.cc:504] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.798781 19478 leader_election.cc:288] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 election: Requested vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:20.799172 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842ae25300854c21b55d97c454fe3b98" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
I20260929 00:50:20.799340 19117 raft_consensus.cc:3037] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.799474 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842ae25300854c21b55d97c454fe3b98" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90"
I20260929 00:50:20.799630 19397 raft_consensus.cc:3037] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.800573 19117 raft_consensus.cc:2454] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 1.
I20260929 00:50:20.800992 19188 leader_election.cc:302] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:20.801203 19478 raft_consensus.cc:2781] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:20.801429 19478 raft_consensus.cc:686] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 1 LEADER]: Becoming Leader. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Running, Role: LEADER
I20260929 00:50:20.801760 19397 raft_consensus.cc:2454] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 1.
I20260929 00:50:20.801795 19478 consensus_queue.cc:237] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.805672 18959 catalog_manager.cc:5495] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b0203088bf47a1a064708429e117bd (127.11.40.194). New cstate: current_term: 1 leader_uuid: "48b0203088bf47a1a064708429e117bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } health_report { overall_health: UNKNOWN } } }
W20260929 00:50:20.831235 19444 tablet.cc:2369] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:20.840976 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.837421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.11.40.194:37153)
0929 00:50:20.837583 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.837588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.837603 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:50:20.837701 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.837704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.837759 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.837874 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:20.838120 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.838639 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.839493 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.839729 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.839838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.840122 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:20.840559 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:20.840690 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.840749 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.840830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:50:20.847083 19117 raft_consensus.cc:1261] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Refusing update from remote peer 48b0203088bf47a1a064708429e117bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:20.847083 19397 raft_consensus.cc:1261] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Refusing update from remote peer 48b0203088bf47a1a064708429e117bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:20.848045 19481 consensus_queue.cc:1035] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:20.848322 19478 consensus_queue.cc:1035] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:50:20.851045 19304 tablet.cc:2369] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:20.851594 19164 tablet.cc:2369] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:20.856217 19486 mvcc.cc:205] Tried to move back new op lower bound from 7334473813381541888 to 7334473813207576576. Current Snapshot: MvccSnapshot[applied={T|T < 7334473813381541888}]
I20260929 00:50:20.856243 19484 mvcc.cc:205] Tried to move back new op lower bound from 7334473813381541888 to 7334473813207576576. Current Snapshot: MvccSnapshot[applied={T|T < 7334473813381541888}]
I20260929 00:50:20.857949 19485 mvcc.cc:205] Tried to move back new op lower bound from 7334473813381541888 to 7334473813207576576. Current Snapshot: MvccSnapshot[applied={T|T < 7334473813381541888}]
I20260929 00:50:20.865072 18959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:20.865283 18959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:20.867851 18959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Sending DeleteTablet for 3 replicas of tablet 842ae25300854c21b55d97c454fe3b98
I20260929 00:50:20.868891 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 842ae25300854c21b55d97c454fe3b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:20 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:60968
I20260929 00:50:20.868932 19377 tablet_service.cc:1514] Processing DeleteTablet for tablet 842ae25300854c21b55d97c454fe3b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:20 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:43758
I20260929 00:50:20.869096 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet 842ae25300854c21b55d97c454fe3b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:20 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:37238
I20260929 00:50:20.869551 19494 tablet_replica.cc:330] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: stopping tablet replica
I20260929 00:50:20.869922 19495 tablet_replica.cc:330] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: stopping tablet replica
I20260929 00:50:20.869959 19494 raft_consensus.cc:2229] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:20.870232 19495 raft_consensus.cc:2229] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:20.870232 18959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:50:20.870605 19495 raft_consensus.cc:2258] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:20.870615 19494 raft_consensus.cc:2258] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 00:50:20.870687 18959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:20.872141 19495 ts_tablet_manager.cc:1905] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:20.872141 19494 ts_tablet_manager.cc:1905] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:20.875175 19494 ts_tablet_manager.cc:1918] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:20.875335 19494 log.cc:1196] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/842ae25300854c21b55d97c454fe3b98
I20260929 00:50:20.875718 19494 ts_tablet_manager.cc:1939] T 842ae25300854c21b55d97c454fe3b98 P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting consensus metadata
I20260929 00:50:20.876307 19495 ts_tablet_manager.cc:1918] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:20.876397 19495 log.cc:1196] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/842ae25300854c21b55d97c454fe3b98
I20260929 00:50:20.876703 19495 ts_tablet_manager.cc:1939] T 842ae25300854c21b55d97c454fe3b98 P 48b0203088bf47a1a064708429e117bd: Deleting consensus metadata
I20260929 00:50:20.877285 18944 catalog_manager.cc:4841] TS 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011): tablet 842ae25300854c21b55d97c454fe3b98 (table test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]) successfully deleted
I20260929 00:50:20.877802 19097 tablet_service.cc:1467] Processing CreateTablet for tablet da5986768f6240668e79f19ab397721d (DEFAULT_TABLE table=test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.878054 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5986768f6240668e79f19ab397721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.878015 19237 tablet_service.cc:1467] Processing CreateTablet for tablet da5986768f6240668e79f19ab397721d (DEFAULT_TABLE table=test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.878270 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5986768f6240668e79f19ab397721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.878706 19377 tablet_service.cc:1467] Processing CreateTablet for tablet da5986768f6240668e79f19ab397721d (DEFAULT_TABLE table=test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:20.878964 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5986768f6240668e79f19ab397721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:20.880641 19472 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Bootstrap starting.
I20260929 00:50:20.881121 19493 tablet_replica.cc:330] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: stopping tablet replica
I20260929 00:50:20.881569 19473 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Bootstrap starting.
I20260929 00:50:20.881687 19471 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Bootstrap starting.
I20260929 00:50:20.881726 19472 tablet_bootstrap.cc:654] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.882534 19473 tablet_bootstrap.cc:654] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.882599 18945 catalog_manager.cc:4841] TS 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153): tablet 842ae25300854c21b55d97c454fe3b98 (table test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]) successfully deleted
I20260929 00:50:20.882715 19471 tablet_bootstrap.cc:654] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:20.881464 19493 raft_consensus.cc:2229] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:20.883363 19493 raft_consensus.cc:2258] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:20.883466 19472 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: No bootstrap required, opened a new log
I20260929 00:50:20.883549 19472 ts_tablet_manager.cc:1397] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:20.884369 19472 raft_consensus.cc:348] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.884486 19472 raft_consensus.cc:374] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.884516 19472 raft_consensus.cc:729] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.884631 19472 consensus_queue.cc:260] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.885201 19493 ts_tablet_manager.cc:1905] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:20.885358 19472 ts_tablet_manager.cc:1428] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:20.886130 19471 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: No bootstrap required, opened a new log
I20260929 00:50:20.886214 19471 ts_tablet_manager.cc:1397] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260929 00:50:20.886752 19471 raft_consensus.cc:348] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.886864 19471 raft_consensus.cc:374] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.886904 19471 raft_consensus.cc:729] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.887041 19471 consensus_queue.cc:260] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.887315 19471 ts_tablet_manager.cc:1428] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:20.887814 19493 ts_tablet_manager.cc:1918] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:20.887920 19493 log.cc:1196] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/842ae25300854c21b55d97c454fe3b98
I20260929 00:50:20.888226 19493 ts_tablet_manager.cc:1939] T 842ae25300854c21b55d97c454fe3b98 P 7d79581e63ba4fe3900d89b44d061f90: Deleting consensus metadata
I20260929 00:50:20.889516 18945 catalog_manager.cc:4841] TS 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507): tablet 842ae25300854c21b55d97c454fe3b98 (table test-table [id=1b9cd9583ca341d5ac88a8d5206ba1cc]) successfully deleted
I20260929 00:50:20.890161 19473 tablet_bootstrap.cc:492] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: No bootstrap required, opened a new log
I20260929 00:50:20.890235 19473 ts_tablet_manager.cc:1397] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 00:50:20.890671 19473 raft_consensus.cc:348] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.890775 19473 raft_consensus.cc:374] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:20.890797 19473 raft_consensus.cc:729] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Initialized, Role: FOLLOWER
I20260929 00:50:20.890976 19473 consensus_queue.cc:260] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.891237 19473 ts_tablet_manager.cc:1428] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:50:20.970755 19477 raft_consensus.cc:482] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:20.970957 19477 raft_consensus.cc:504] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.971808 19481 raft_consensus.cc:482] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:20.971930 19481 raft_consensus.cc:504] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.972359 19481 leader_election.cc:288] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:20.972344 19477 leader_election.cc:288] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:20.973003 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90" is_pre_election: true
I20260929 00:50:20.973124 19397 raft_consensus.cc:2454] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 0.
I20260929 00:50:20.973445 19189 leader_election.cc:302] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 7d79581e63ba4fe3900d89b44d061f90; no voters:
I20260929 00:50:20.973703 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" is_pre_election: true
I20260929 00:50:20.973867 19117 raft_consensus.cc:2454] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 0.
I20260929 00:50:20.974304 19481 raft_consensus.cc:2781] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:20.974365 19481 raft_consensus.cc:482] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:20.974402 19481 raft_consensus.cc:3037] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.976022 19481 raft_consensus.cc:504] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.976430 19481 leader_election.cc:288] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 election: Requested vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:20.977042 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
I20260929 00:50:20.977216 19117 raft_consensus.cc:3037] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.977293 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "48b0203088bf47a1a064708429e117bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90"
I20260929 00:50:20.977419 19397 raft_consensus.cc:3037] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:20.978215 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.972960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:54357 (local address 127.11.40.194:37153)
0929 00:50:20.973080 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.973085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.973367 (+ 282us) server_negotiation.cc:407] Connection header received
0929 00:50:20.973556 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.973559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.973607 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.973654 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.974127 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.975018 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.976875 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.977429 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.977533 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.977609 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.978013 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.978099 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:50:20.978416 19117 raft_consensus.cc:2454] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 1.
I20260929 00:50:20.978636 19397 raft_consensus.cc:2454] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0203088bf47a1a064708429e117bd in term 1.
I20260929 00:50:20.978675 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.972599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:37153 (local address 127.11.40.195:54357)
0929 00:50:20.973289 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.973302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.973433 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.973865 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.973868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.973880 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.974006 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.974014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.975269 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.975272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.976702 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.976713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.977747 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.977750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.977853 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.977894 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.978389 (+ 495us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.978469 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":88,"threads_started":1}
I20260929 00:50:20.978801 19188 leader_election.cc:302] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:20.979238 19481 raft_consensus.cc:2781] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:20.979267 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd" is_pre_election: true
I20260929 00:50:20.979604 19481 raft_consensus.cc:686] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 1 LEADER]: Becoming Leader. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Running, Role: LEADER
I20260929 00:50:20.979779 19481 consensus_queue.cc:237] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:20.980127 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.972571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43011 (local address 127.11.40.195:35513)
0929 00:50:20.972895 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:20.972912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:20.973017 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:20.973789 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:20.973795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:20.973813 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:20.973936 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.973944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.975049 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.975054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.976646 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:20.976658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.979676 (+ 3018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.979680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:20.979793 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.979820 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:50:20.979927 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 00:50:20.980007 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":141}
I20260929 00:50:20.980170 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:20.973199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:35513 (local address 127.11.40.193:43011)
0929 00:50:20.973515 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:20.973522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:20.973536 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:20.973577 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:20.973580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:20.973622 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:20.973690 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:20.974064 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.974896 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.976894 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:20.979535 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:20.979652 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:20.979705 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:20.979968 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 00:50:20.980052 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260929 00:50:20.980686 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5986768f6240668e79f19ab397721d" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" is_pre_election: true
I20260929 00:50:20.980913 19117 raft_consensus.cc:2379] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d79581e63ba4fe3900d89b44d061f90 in current term 1: Already voted for candidate 48b0203088bf47a1a064708429e117bd in this term.
I20260929 00:50:20.981492 19329 leader_election.cc:302] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d79581e63ba4fe3900d89b44d061f90; no voters: 48b0203088bf47a1a064708429e117bd, 8d2f50e746b94bddbe66fd9fbb16c4b8
I20260929 00:50:20.981782 19477 raft_consensus.cc:2726] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 00:50:20.982268 18954 catalog_manager.cc:5495] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b0203088bf47a1a064708429e117bd (127.11.40.194). New cstate: current_term: 1 leader_uuid: "48b0203088bf47a1a064708429e117bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:50:21 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643019, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.254@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:21.035907 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.026739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42451 (local address 127.0.0.1:40666)
0929 00:50:21.027128 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.027141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.027295 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.027562 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.027565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.027993 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:21.028089 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.028095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.028912 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.028917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.029506 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.029514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.029964 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.029967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.030045 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.030674 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:21.030687 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:21.032334 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:21.034218 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:21.034230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:21.034237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:21.034494 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:21.034778 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:21.034781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:21.034783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:21.034864 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:21.035278 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:21.035287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:21.035404 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.035523 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.035693 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":162,"threads_started":1}
I20260929 00:50:21.035907 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.026875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.11.40.254:42451)
0929 00:50:21.027035 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.027040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.027231 (+ 191us) server_negotiation.cc:407] Connection header received
0929 00:50:21.027401 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.027404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.027449 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.027540 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:21.028217 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.028752 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.029631 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.029849 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.029959 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.032510 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:21.032513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:21.032515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:21.032536 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:21.034104 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:21.034598 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:21.034603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:21.034605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:21.034683 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:21.034960 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:21.034965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:21.034967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:21.035162 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:21.035268 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.035584 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.035776 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":64}
I20260929 00:50:21.040619 18954 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:40666:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:50:21.041503 18954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:21.046936 19237 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9f520ac2954541b9e19de2c18b966c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72fcd05dec614291880993dec14a57d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:50:21.046950 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9f520ac2954541b9e19de2c18b966c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72fcd05dec614291880993dec14a57d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:50:21.047016 19377 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9f520ac2954541b9e19de2c18b966c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72fcd05dec614291880993dec14a57d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:50:21.047369 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9f520ac2954541b9e19de2c18b966c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.047363 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9f520ac2954541b9e19de2c18b966c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.047425 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9f520ac2954541b9e19de2c18b966c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.050148 19473 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: Bootstrap starting.
I20260929 00:50:21.050295 19472 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Bootstrap starting.
I20260929 00:50:21.050454 19471 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: Bootstrap starting.
I20260929 00:50:21.051216 19473 tablet_bootstrap.cc:654] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.051273 19472 tablet_bootstrap.cc:654] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.051316 19471 tablet_bootstrap.cc:654] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.052820 19473 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: No bootstrap required, opened a new log
I20260929 00:50:21.052906 19473 ts_tablet_manager.cc:1397] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:21.053228 19471 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: No bootstrap required, opened a new log
I20260929 00:50:21.053303 19471 ts_tablet_manager.cc:1397] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 00:50:21.053622 19473 raft_consensus.cc:348] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.053678 19471 raft_consensus.cc:348] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.053750 19473 raft_consensus.cc:374] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.053789 19471 raft_consensus.cc:374] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.053797 19473 raft_consensus.cc:729] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.053831 19471 raft_consensus.cc:729] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.053946 19473 consensus_queue.cc:260] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.053970 19471 consensus_queue.cc:260] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.054255 19471 ts_tablet_manager.cc:1428] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 00:50:21.054303 19473 ts_tablet_manager.cc:1428] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:21.054253 19477 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.054399 19477 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.054441 19481 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.054550 19481 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.055190 19472 tablet_bootstrap.cc:492] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: No bootstrap required, opened a new log
I20260929 00:50:21.055282 19472 ts_tablet_manager.cc:1397] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 00:50:21.055702 19472 raft_consensus.cc:348] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.055848 19472 raft_consensus.cc:374] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.055889 19472 raft_consensus.cc:729] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.056021 19472 consensus_queue.cc:260] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.056360 19479 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.056509 19479 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.056376 19472 ts_tablet_manager.cc:1428] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:21.158087 19477 raft_consensus.cc:482] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:21.158241 19477 raft_consensus.cc:504] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.158762 19477 leader_election.cc:288] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:21.159212 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9f520ac2954541b9e19de2c18b966c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" is_pre_election: true
I20260929 00:50:21.159277 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9f520ac2954541b9e19de2c18b966c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd" is_pre_election: true
I20260929 00:50:21.159366 19117 raft_consensus.cc:2454] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 0.
I20260929 00:50:21.159543 19255 raft_consensus.cc:2454] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 0.
I20260929 00:50:21.159736 19328 leader_election.cc:302] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d79581e63ba4fe3900d89b44d061f90, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:21.159940 19477 raft_consensus.cc:2781] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:21.160003 19477 raft_consensus.cc:482] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:21.160050 19477 raft_consensus.cc:3037] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.161176 19477 raft_consensus.cc:504] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.161558 19477 leader_election.cc:288] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 election: Requested vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:21.161938 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9f520ac2954541b9e19de2c18b966c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd"
I20260929 00:50:21.161978 19117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9f520ac2954541b9e19de2c18b966c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
I20260929 00:50:21.162060 19255 raft_consensus.cc:3037] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.162113 19117 raft_consensus.cc:3037] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.163028 19117 raft_consensus.cc:2454] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 1.
I20260929 00:50:21.163028 19255 raft_consensus.cc:2454] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 1.
I20260929 00:50:21.163306 19329 leader_election.cc:302] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 7d79581e63ba4fe3900d89b44d061f90; no voters:
I20260929 00:50:21.163499 19477 raft_consensus.cc:2781] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:21.163705 19477 raft_consensus.cc:686] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 LEADER]: Becoming Leader. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Running, Role: LEADER
I20260929 00:50:21.163976 19477 consensus_queue.cc:237] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.165210 19510 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: TxnStatusTablet state changed. Reason: New leader 7d79581e63ba4fe3900d89b44d061f90. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.165338 19510 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:50:21.165570 19512 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:50:21.165729 19512 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:50:21.166571 18960 catalog_manager.cc:5495] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } health_report { overall_health: HEALTHY } } }
I20260929 00:50:21.179781 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.175223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.0.0.1:43764)
0929 00:50:21.175382 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.175400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.175505 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.175914 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.175916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.175923 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:50:21.176051 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.176057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.177149 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.177151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.177773 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.177781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.178447 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.178452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.178550 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.178602 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.179479 (+ 877us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.179498 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.179567 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.179655 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 00:50:21.179836 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.175299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.11.40.195:41507)
0929 00:50:21.175442 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.175447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.175463 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:21.175669 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.175673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.175743 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.175852 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.176171 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.177044 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.177890 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.178226 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.178340 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.178755 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.179318 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.179473 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.179584 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.179659 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:50:21.183429 19117 raft_consensus.cc:1261] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Refusing update from remote peer 7d79581e63ba4fe3900d89b44d061f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.183611 19255 raft_consensus.cc:1261] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Refusing update from remote peer 7d79581e63ba4fe3900d89b44d061f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.184092 19510 consensus_queue.cc:1035] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.184355 19477 consensus_queue.cc:1035] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.186262 19479 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: TxnStatusTablet state changed. Reason: New leader 7d79581e63ba4fe3900d89b44d061f90. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.186414 19479 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.186537 19481 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: TxnStatusTablet state changed. Reason: New leader 7d79581e63ba4fe3900d89b44d061f90. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.186658 19481 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.189625 19477 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.189757 19477 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:50:21.190003 19479 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.190119 19479 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 8d2f50e746b94bddbe66fd9fbb16c4b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.191857 19510 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.191975 19510 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 7d79581e63ba4fe3900d89b44d061f90: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:50:21.192667 19481 tablet_replica.cc:439] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } } }
I20260929 00:50:21.192783 19481 tablet_replica.cc:442] T 0f9f520ac2954541b9e19de2c18b966c P 48b0203088bf47a1a064708429e117bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:50:21.204751 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.201258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.11.40.194:54335)
0929 00:50:21.201584 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.201599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.201739 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.202008 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.202011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.202019 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:50:21.202137 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.202143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.202875 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.202880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.203506 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.203514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.203983 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.203987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.204087 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.204129 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.204450 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.204459 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.204525 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.204610 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
I20260929 00:50:21.204749 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.201422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:54335 (local address 127.11.40.195:41507)
0929 00:50:21.201555 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.201560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.201678 (+ 118us) server_negotiation.cc:407] Connection header received
0929 00:50:21.201845 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.201848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.201907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.201988 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.202241 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.202759 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.203635 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.203872 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.203981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.204271 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.204368 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.204447 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.204534 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.204585 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:50:21.211725 19397 raft_consensus.cc:1261] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Refusing update from remote peer 48b0203088bf47a1a064708429e117bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.211992 19117 raft_consensus.cc:1261] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Refusing update from remote peer 48b0203088bf47a1a064708429e117bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.212177 19481 consensus_queue.cc:1035] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.212325 19478 consensus_queue.cc:1035] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.236617 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.232601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52675 (local address 127.11.40.194:37153)
0929 00:50:21.232722 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.232728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.232959 (+ 231us) server_negotiation.cc:407] Connection header received
0929 00:50:21.233067 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.233070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.233123 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.233203 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.233551 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.234334 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.235291 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.235641 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.235753 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.236170 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.236249 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.236325 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.236437 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.236496 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:50:21.236922 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.232146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:37153 (local address 127.11.40.195:52675)
0929 00:50:21.232857 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.232874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.232989 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.233246 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.233249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.233256 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:50:21.233434 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.233440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.234464 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.234470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.235152 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.235162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.235792 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.235796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.235910 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.235972 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.236345 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.236352 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.236417 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.236514 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":138,"threads_started":1}
I20260929 00:50:21.259423 18954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.259651 18954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.262029 18954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Sending DeleteTablet for 3 replicas of tablet da5986768f6240668e79f19ab397721d
I20260929 00:50:21.262740 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet da5986768f6240668e79f19ab397721d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:60968
I20260929 00:50:21.263042 19494 tablet_replica.cc:330] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: stopping tablet replica
I20260929 00:50:21.263130 19377 tablet_service.cc:1514] Processing DeleteTablet for tablet da5986768f6240668e79f19ab397721d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:43758
I20260929 00:50:21.263178 19494 raft_consensus.cc:2229] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.263312 19493 tablet_replica.cc:330] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: stopping tablet replica
I20260929 00:50:21.263314 19494 raft_consensus.cc:2258] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.263417 19493 raft_consensus.cc:2229] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.263540 19493 raft_consensus.cc:2258] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.263705 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet da5986768f6240668e79f19ab397721d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:37238
I20260929 00:50:21.264137 19495 tablet_replica.cc:330] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: stopping tablet replica
I20260929 00:50:21.264370 19495 raft_consensus.cc:2229] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:21.264628 19495 raft_consensus.cc:2258] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.265678 19493 ts_tablet_manager.cc:1905] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.269989 19495 ts_tablet_manager.cc:1905] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.270037 19493 ts_tablet_manager.cc:1918] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:50:21.270112 19493 log.cc:1196] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/da5986768f6240668e79f19ab397721d
I20260929 00:50:21.270381 19493 ts_tablet_manager.cc:1939] T da5986768f6240668e79f19ab397721d P 7d79581e63ba4fe3900d89b44d061f90: Deleting consensus metadata
I20260929 00:50:21.271265 18945 catalog_manager.cc:4841] TS 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507): tablet da5986768f6240668e79f19ab397721d (table test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]) successfully deleted
I20260929 00:50:21.271746 19494 ts_tablet_manager.cc:1905] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.273036 19495 ts_tablet_manager.cc:1918] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:50:21.273126 19495 log.cc:1196] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/da5986768f6240668e79f19ab397721d
I20260929 00:50:21.273409 19495 ts_tablet_manager.cc:1939] T da5986768f6240668e79f19ab397721d P 48b0203088bf47a1a064708429e117bd: Deleting consensus metadata
I20260929 00:50:21.274278 18945 catalog_manager.cc:4841] TS 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153): tablet da5986768f6240668e79f19ab397721d (table test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]) successfully deleted
I20260929 00:50:21.274751 19494 ts_tablet_manager.cc:1918] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:50:21.274819 19494 log.cc:1196] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/da5986768f6240668e79f19ab397721d
I20260929 00:50:21.275115 19494 ts_tablet_manager.cc:1939] T da5986768f6240668e79f19ab397721d P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting consensus metadata
I20260929 00:50:21.275933 18944 catalog_manager.cc:4841] TS 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011): tablet da5986768f6240668e79f19ab397721d (table test-table [id=5e3c508994054445b1b21fc9c5d4b1f9]) successfully deleted
Sep 29 00:50:21 dist-test-slave-5309 krb5kdc[18915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:50:21 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:21 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:50:21.300289 19009 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:21.292738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.11.40.254:42451)
0929 00:50:21.292946 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.292951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.293133 (+ 182us) server_negotiation.cc:407] Connection header received
0929 00:50:21.293264 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.293269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.293324 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.293420 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:21.294225 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.295076 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.296023 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.296381 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.296507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.300148 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40668: BlockingRecv error: recv got EOF from 127.0.0.1:40668 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 00:50:21 dist-test-slave-5309 krb5kdc[18915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM
I20260929 00:50:21.312248 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.302232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.11.40.254:42451)
0929 00:50:21.302347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.302352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.302556 (+ 204us) server_negotiation.cc:407] Connection header received
0929 00:50:21.302660 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.302664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.302716 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.302807 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:21.303402 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.304212 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.305105 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.305461 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.305567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.308342 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:21.308347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:21.308350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:21.308384 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:21.310275 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:21.310775 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:21.310780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:21.310782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:21.310886 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:21.311210 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:21.311213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:21.311215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:21.311490 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:21.311636 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.311929 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.312069 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:50:21.315341 18954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:21.315828 18954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:21.320694 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7509ab58874d94ad063bed8702cf1b (DEFAULT_TABLE table=test-table [id=bbfc0d02f96445e79c45d10a57c50f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.320955 19237 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7509ab58874d94ad063bed8702cf1b (DEFAULT_TABLE table=test-table [id=bbfc0d02f96445e79c45d10a57c50f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.320979 19377 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7509ab58874d94ad063bed8702cf1b (DEFAULT_TABLE table=test-table [id=bbfc0d02f96445e79c45d10a57c50f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.321089 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7509ab58874d94ad063bed8702cf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.321218 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7509ab58874d94ad063bed8702cf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.321292 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7509ab58874d94ad063bed8702cf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.323607 19471 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Bootstrap starting.
I20260929 00:50:21.323907 19473 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Bootstrap starting.
I20260929 00:50:21.324569 19471 tablet_bootstrap.cc:654] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.325066 19473 tablet_bootstrap.cc:654] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.325948 19471 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: No bootstrap required, opened a new log
I20260929 00:50:21.326063 19471 ts_tablet_manager.cc:1397] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 00:50:21.326261 19473 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: No bootstrap required, opened a new log
I20260929 00:50:21.326354 19473 ts_tablet_manager.cc:1397] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:50:21.326500 19471 raft_consensus.cc:348] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.326628 19471 raft_consensus.cc:374] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.326668 19471 raft_consensus.cc:729] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.326809 19471 consensus_queue.cc:260] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.327098 19471 ts_tablet_manager.cc:1428] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 00:50:21.327162 19473 raft_consensus.cc:348] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.327286 19473 raft_consensus.cc:374] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.327322 19473 raft_consensus.cc:729] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.327445 19472 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Bootstrap starting.
I20260929 00:50:21.327489 19473 consensus_queue.cc:260] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.327942 19473 ts_tablet_manager.cc:1428] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:50:21.328368 19472 tablet_bootstrap.cc:654] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.333163 19472 tablet_bootstrap.cc:492] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: No bootstrap required, opened a new log
I20260929 00:50:21.333271 19472 ts_tablet_manager.cc:1397] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260929 00:50:21.333719 19472 raft_consensus.cc:348] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.333846 19472 raft_consensus.cc:374] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.333874 19472 raft_consensus.cc:729] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.333992 19472 consensus_queue.cc:260] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.334905 19472 ts_tablet_manager.cc:1428] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:50:21.408293 19479 raft_consensus.cc:482] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:21.408464 19479 raft_consensus.cc:504] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.409597 19479 leader_election.cc:288] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:21.414191 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.410072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37177 (local address 127.11.40.194:37153)
0929 00:50:21.410248 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.410253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.410410 (+ 157us) server_negotiation.cc:407] Connection header received
0929 00:50:21.410650 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.410658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.410739 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.410883 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:21.411232 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.411963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.413155 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.413571 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.413707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.413768 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.414007 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.414079 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:50:21.414215 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.409984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:37153 (local address 127.11.40.193:37177)
0929 00:50:21.410303 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.410321 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.410474 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.410941 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.410944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.410970 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:21.411116 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.411127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.412081 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.412087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.413027 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.413036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.413699 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.413702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.413851 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.413870 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.413972 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.414047 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":24,"thread_start_us":109,"threads_started":1}
I20260929 00:50:21.414852 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7509ab58874d94ad063bed8702cf1b" candidate_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd" is_pre_election: true
I20260929 00:50:21.415050 19255 raft_consensus.cc:2454] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f50e746b94bddbe66fd9fbb16c4b8 in term 0.
I20260929 00:50:21.415203 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.410575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59699 (local address 127.11.40.195:41507)
0929 00:50:21.410699 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.410704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.411490 (+ 786us) server_negotiation.cc:407] Connection header received
0929 00:50:21.411584 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.411587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.411642 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.411726 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:21.412125 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.412719 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.413921 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.414672 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.414830 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.414894 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.414991 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.415073 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:50:21.415288 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.411023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.11.40.193:59699)
0929 00:50:21.411389 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.411402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.411497 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.411783 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.411786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.411809 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:21.411940 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.411946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.412912 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.412915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.413789 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.413797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.414809 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.414812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.414890 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.414908 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.414969 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.415114 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260929 00:50:21.415493 19049 leader_election.cc:302] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:21.415731 19479 raft_consensus.cc:2781] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:21.415789 19479 raft_consensus.cc:482] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:21.415838 19479 raft_consensus.cc:3037] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.416085 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7509ab58874d94ad063bed8702cf1b" candidate_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90" is_pre_election: true
I20260929 00:50:21.416396 19397 raft_consensus.cc:2454] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f50e746b94bddbe66fd9fbb16c4b8 in term 0.
I20260929 00:50:21.417176 19479 raft_consensus.cc:504] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.417534 19479 leader_election.cc:288] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153), 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507)
I20260929 00:50:21.418001 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7509ab58874d94ad063bed8702cf1b" candidate_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d79581e63ba4fe3900d89b44d061f90"
I20260929 00:50:21.418112 19397 raft_consensus.cc:3037] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.418041 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7509ab58874d94ad063bed8702cf1b" candidate_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd"
I20260929 00:50:21.418223 19255 raft_consensus.cc:3037] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.419047 19397 raft_consensus.cc:2454] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f50e746b94bddbe66fd9fbb16c4b8 in term 1.
I20260929 00:50:21.419373 19049 leader_election.cc:302] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d79581e63ba4fe3900d89b44d061f90, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:21.419420 19255 raft_consensus.cc:2454] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f50e746b94bddbe66fd9fbb16c4b8 in term 1.
I20260929 00:50:21.419589 19479 raft_consensus.cc:2781] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:21.419859 19479 raft_consensus.cc:686] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Running, Role: LEADER
I20260929 00:50:21.420147 19479 consensus_queue.cc:237] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.422827 18954 catalog_manager.cc:5495] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } health_report { overall_health: UNKNOWN } } }
I20260929 00:50:21.439774 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.435952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.11.40.193:43011)
0929 00:50:21.436094 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.436098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.436114 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:21.436206 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.436208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.436285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.436397 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.436710 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.437386 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.438228 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.438495 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.438587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.438871 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.439368 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.439509 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.439578 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.439636 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:50:21.442886 19397 raft_consensus.cc:1261] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Refusing update from remote peer 8d2f50e746b94bddbe66fd9fbb16c4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.442936 19255 raft_consensus.cc:1261] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Refusing update from remote peer 8d2f50e746b94bddbe66fd9fbb16c4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.443531 19479 consensus_queue.cc:1035] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.443764 19544 consensus_queue.cc:1035] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.455911 18959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.456108 18959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.458495 18959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Sending DeleteTablet for 3 replicas of tablet 1f7509ab58874d94ad063bed8702cf1b
I20260929 00:50:21.459136 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7509ab58874d94ad063bed8702cf1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:60968
I20260929 00:50:21.459332 19494 tablet_replica.cc:330] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: stopping tablet replica
I20260929 00:50:21.459322 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7509ab58874d94ad063bed8702cf1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:37238
I20260929 00:50:21.459466 19494 raft_consensus.cc:2229] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:21.459533 19495 tablet_replica.cc:330] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: stopping tablet replica
I20260929 00:50:21.459679 19495 raft_consensus.cc:2229] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.459791 19494 raft_consensus.cc:2258] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.459821 19495 raft_consensus.cc:2258] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.459877 18959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:21.460323 18959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:21.460369 19377 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7509ab58874d94ad063bed8702cf1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:43758
I20260929 00:50:21.460570 19493 tablet_replica.cc:330] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: stopping tablet replica
I20260929 00:50:21.460582 19495 ts_tablet_manager.cc:1905] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.460675 19493 raft_consensus.cc:2229] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.460887 19493 raft_consensus.cc:2258] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.461694 19493 ts_tablet_manager.cc:1905] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.463128 19495 ts_tablet_manager.cc:1918] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:21.463186 19495 log.cc:1196] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/1f7509ab58874d94ad063bed8702cf1b
I20260929 00:50:21.463397 19495 ts_tablet_manager.cc:1939] T 1f7509ab58874d94ad063bed8702cf1b P 48b0203088bf47a1a064708429e117bd: Deleting consensus metadata
I20260929 00:50:21.464118 18945 catalog_manager.cc:4841] TS 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153): tablet 1f7509ab58874d94ad063bed8702cf1b (table test-table [id=bbfc0d02f96445e79c45d10a57c50f79]) successfully deleted
I20260929 00:50:21.464231 19493 ts_tablet_manager.cc:1918] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:21.464303 19493 log.cc:1196] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/1f7509ab58874d94ad063bed8702cf1b
I20260929 00:50:21.464605 19493 ts_tablet_manager.cc:1939] T 1f7509ab58874d94ad063bed8702cf1b P 7d79581e63ba4fe3900d89b44d061f90: Deleting consensus metadata
I20260929 00:50:21.465502 19494 ts_tablet_manager.cc:1905] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:21.465652 18945 catalog_manager.cc:4841] TS 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507): tablet 1f7509ab58874d94ad063bed8702cf1b (table test-table [id=bbfc0d02f96445e79c45d10a57c50f79]) successfully deleted
I20260929 00:50:21.468027 19494 ts_tablet_manager.cc:1918] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:21.468128 19494 log.cc:1196] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/1f7509ab58874d94ad063bed8702cf1b
I20260929 00:50:21.468319 19494 ts_tablet_manager.cc:1939] T 1f7509ab58874d94ad063bed8702cf1b P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting consensus metadata
I20260929 00:50:21.469201 19237 tablet_service.cc:1467] Processing CreateTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c (DEFAULT_TABLE table=test-table [id=67c0b9c198a64c9fb6b27fa5d0d109de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.469238 19377 tablet_service.cc:1467] Processing CreateTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c (DEFAULT_TABLE table=test-table [id=67c0b9c198a64c9fb6b27fa5d0d109de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.469298 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c (DEFAULT_TABLE table=test-table [id=67c0b9c198a64c9fb6b27fa5d0d109de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:21.469493 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c513615cbb94afd842d52b2fcdf5d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.469493 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c513615cbb94afd842d52b2fcdf5d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.469552 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c513615cbb94afd842d52b2fcdf5d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:21.471794 19472 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Bootstrap starting.
I20260929 00:50:21.471807 19473 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Bootstrap starting.
I20260929 00:50:21.472675 19473 tablet_bootstrap.cc:654] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.472723 19472 tablet_bootstrap.cc:654] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.472952 19471 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Bootstrap starting.
I20260929 00:50:21.473734 19471 tablet_bootstrap.cc:654] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:21.473763 19473 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: No bootstrap required, opened a new log
I20260929 00:50:21.473822 19473 ts_tablet_manager.cc:1397] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:50:21.474169 19473 raft_consensus.cc:348] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.474298 19473 raft_consensus.cc:374] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.474332 19473 raft_consensus.cc:729] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.474490 19473 consensus_queue.cc:260] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.474758 19473 ts_tablet_manager.cc:1428] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:21.474844 18944 catalog_manager.cc:4841] TS 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011): tablet 1f7509ab58874d94ad063bed8702cf1b (table test-table [id=bbfc0d02f96445e79c45d10a57c50f79]) successfully deleted
I20260929 00:50:21.476346 19471 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: No bootstrap required, opened a new log
I20260929 00:50:21.476469 19471 ts_tablet_manager.cc:1397] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 00:50:21.476895 19471 raft_consensus.cc:348] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.477018 19471 raft_consensus.cc:374] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.477057 19471 raft_consensus.cc:729] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0203088bf47a1a064708429e117bd, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.477181 19471 consensus_queue.cc:260] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.477396 19472 tablet_bootstrap.cc:492] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: No bootstrap required, opened a new log
I20260929 00:50:21.477443 19471 ts_tablet_manager.cc:1428] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:50:21.477521 19472 ts_tablet_manager.cc:1397] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260929 00:50:21.477995 19472 raft_consensus.cc:348] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.478139 19472 raft_consensus.cc:374] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:21.478183 19472 raft_consensus.cc:729] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f50e746b94bddbe66fd9fbb16c4b8, State: Initialized, Role: FOLLOWER
I20260929 00:50:21.478310 19472 consensus_queue.cc:260] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.478592 19472 ts_tablet_manager.cc:1428] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 00:50:21.843683 19477 raft_consensus.cc:482] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:21.843827 19477 raft_consensus.cc:504] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.844269 19477 leader_election.cc:288] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:21.844719 19116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c513615cbb94afd842d52b2fcdf5d6c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" is_pre_election: true
I20260929 00:50:21.844744 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c513615cbb94afd842d52b2fcdf5d6c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd" is_pre_election: true
I20260929 00:50:21.844933 19255 raft_consensus.cc:2454] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 0.
I20260929 00:50:21.844944 19116 raft_consensus.cc:2454] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 0.
I20260929 00:50:21.845274 19329 leader_election.cc:302] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0203088bf47a1a064708429e117bd, 7d79581e63ba4fe3900d89b44d061f90; no voters:
I20260929 00:50:21.845503 19477 raft_consensus.cc:2781] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:21.845578 19477 raft_consensus.cc:482] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:21.845630 19477 raft_consensus.cc:3037] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.846690 19477 raft_consensus.cc:504] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.847067 19477 leader_election.cc:288] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 election: Requested vote from peers 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011), 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153)
I20260929 00:50:21.847463 19116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c513615cbb94afd842d52b2fcdf5d6c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8"
I20260929 00:50:21.847468 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c513615cbb94afd842d52b2fcdf5d6c" candidate_uuid: "7d79581e63ba4fe3900d89b44d061f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0203088bf47a1a064708429e117bd"
I20260929 00:50:21.847564 19116 raft_consensus.cc:3037] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.847604 19255 raft_consensus.cc:3037] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:21.848543 19116 raft_consensus.cc:2454] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 1.
I20260929 00:50:21.848559 19255 raft_consensus.cc:2454] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d79581e63ba4fe3900d89b44d061f90 in term 1.
I20260929 00:50:21.848923 19328 leader_election.cc:302] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d79581e63ba4fe3900d89b44d061f90, 8d2f50e746b94bddbe66fd9fbb16c4b8; no voters:
I20260929 00:50:21.849123 19477 raft_consensus.cc:2781] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:21.849223 19477 raft_consensus.cc:686] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 LEADER]: Becoming Leader. State: Replica: 7d79581e63ba4fe3900d89b44d061f90, State: Running, Role: LEADER
I20260929 00:50:21.849424 19477 consensus_queue.cc:237] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } }
I20260929 00:50:21.851223 18954 catalog_manager.cc:5495] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "7d79581e63ba4fe3900d89b44d061f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d79581e63ba4fe3900d89b44d061f90" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41507 } health_report { overall_health: HEALTHY } } }
I20260929 00:50:21.939956 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.936702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.11.40.195:41507)
0929 00:50:21.936961 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.936965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.936981 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:21.937035 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.937038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.937094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.937192 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.937440 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.937997 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.938854 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.939094 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.939208 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.939471 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.939557 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.939642 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.939733 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.939787 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":56,"threads_started":1}
I20260929 00:50:21.949635 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.946203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:35231 (local address 127.11.40.195:41507)
0929 00:50:21.946364 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:21.946368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:21.946461 (+ 93us) server_negotiation.cc:407] Connection header received
0929 00:50:21.946552 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:21.946555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:21.946596 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:21.946659 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:21.946868 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.947426 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.948441 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.948892 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.948995 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.949220 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.949291 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.949362 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:21.949441 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 00:50:21.949499 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":95}
I20260929 00:50:21.949644 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:21.946105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41507 (local address 127.11.40.195:35231)
0929 00:50:21.946370 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:21.946386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:21.946486 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:21.946684 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:21.946686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:21.946693 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:50:21.946777 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.946781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.947551 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.947553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.948209 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:21.948218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:21.949016 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:21.949019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:21.949085 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:21.949112 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:50:21.949378 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:50:21.949383 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:50:21.949432 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 00:50:21.949505 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":119,"threads_started":1}
I20260929 00:50:21.955667 19116 raft_consensus.cc:1261] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Refusing update from remote peer 7d79581e63ba4fe3900d89b44d061f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.955770 19255 raft_consensus.cc:1261] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Refusing update from remote peer 7d79581e63ba4fe3900d89b44d061f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:21.956090 19554 consensus_queue.cc:1035] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f50e746b94bddbe66fd9fbb16c4b8" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.956297 19477 consensus_queue.cc:1035] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b0203088bf47a1a064708429e117bd" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:21.992619 18954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.992787 18954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:21.995129 18954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72c1bd56abd14ed484918a1317f9e180: Sending DeleteTablet for 3 replicas of tablet 8c513615cbb94afd842d52b2fcdf5d6c
I20260929 00:50:21.995781 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:60968
I20260929 00:50:21.995921 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:37238
I20260929 00:50:21.996106 19565 tablet_replica.cc:330] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: stopping tablet replica
I20260929 00:50:21.996238 19565 raft_consensus.cc:2229] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.996362 19565 raft_consensus.cc:2258] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.996346 19377 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c513615cbb94afd842d52b2fcdf5d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:21 UTC) from {username='oryx', principal='oryx/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:43758
I20260929 00:50:21.996718 19566 tablet_replica.cc:330] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: stopping tablet replica
I20260929 00:50:21.996898 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19032
I20260929 00:50:21.996901 19566 raft_consensus.cc:2229] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:21.997042 19566 raft_consensus.cc:2258] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:21.997062 19565 ts_tablet_manager.cc:1905] T 8c513615cbb94afd842d52b2fcdf5d6c P 8d2f50e746b94bddbe66fd9fbb16c4b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:22.000202 19567 tablet_replica.cc:330] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: stopping tablet replica
I20260929 00:50:22.000439 19567 raft_consensus.cc:2229] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:22.001209 19566 ts_tablet_manager.cc:1905] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:22.005144 19566 ts_tablet_manager.cc:1918] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:50:22.005261 19566 log.cc:1196] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/8c513615cbb94afd842d52b2fcdf5d6c
I20260929 00:50:22.005585 19566 ts_tablet_manager.cc:1939] T 8c513615cbb94afd842d52b2fcdf5d6c P 48b0203088bf47a1a064708429e117bd: Deleting consensus metadata
I20260929 00:50:22.005728 19567 raft_consensus.cc:2258] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:22.006430 19567 ts_tablet_manager.cc:1905] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:22.006806 18945 catalog_manager.cc:4841] TS 48b0203088bf47a1a064708429e117bd (127.11.40.194:37153): tablet 8c513615cbb94afd842d52b2fcdf5d6c (table test-table [id=67c0b9c198a64c9fb6b27fa5d0d109de]) successfully deleted
W20260929 00:50:22.007939 18944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:43011 (error 108)
I20260929 00:50:22.008296 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19172
I20260929 00:50:22.010151 19567 ts_tablet_manager.cc:1918] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:50:22.010293 19567 log.cc:1196] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/8c513615cbb94afd842d52b2fcdf5d6c
I20260929 00:50:22.010587 19567 ts_tablet_manager.cc:1939] T 8c513615cbb94afd842d52b2fcdf5d6c P 7d79581e63ba4fe3900d89b44d061f90: Deleting consensus metadata
I20260929 00:50:22.011412 18945 catalog_manager.cc:4841] TS 7d79581e63ba4fe3900d89b44d061f90 (127.11.40.195:41507): tablet 8c513615cbb94afd842d52b2fcdf5d6c (table test-table [id=67c0b9c198a64c9fb6b27fa5d0d109de]) successfully deleted
I20260929 00:50:22.011544 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:22.009503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43011 (local address 127.0.0.1:60978)
0929 00:50:22.011356 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:22.011440 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:43011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1767,"thread_start_us":63,"threads_started":1}
W20260929 00:50:22.011706 18944 catalog_manager.cc:4584] TS 8d2f50e746b94bddbe66fd9fbb16c4b8 (127.11.40.193:43011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c513615cbb94afd842d52b2fcdf5d6c: Network error: Client connection negotiation failed: client connection to 127.11.40.193:43011: connect: Connection refused (error 111)
I20260929 00:50:22.017490 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19312
I20260929 00:50:22.024550 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 18928
2026-09-29T00:50:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3616 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:22 dist-test-slave-5309 krb5kdc[19572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:22 dist-test-slave-5309 krb5kdc[19572](info): set up 2 sockets
Sep 29 00:50:22 dist-test-slave-5309 krb5kdc[19572](info): commencing operation
krb5kdc: starting...
W20260929 00:50:23.091624 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:23 dist-test-slave-5309 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:50:23.220077 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42655
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:23.338713 19585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:23.339003 19585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:23.339047 19585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:23.343065 19585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:23.343142 19585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:23.343165 19585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:23.343185 19585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:23.343204 19585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:23.348901 19585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:23.350013 19585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:23.351034 19585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:23.360208 19591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:23.360208 19590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:23.360358 19593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:23.360414 19585 server_base.cc:1034] running on GCE node
I20260929 00:50:23.361222 19585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:23.362382 19585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:23.363587 19585 hybrid_clock.cc:648] HybridClock initialized: now 1790643023363567 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T00:50:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1352 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:23 dist-test-slave-5309 krb5kdc[19598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:23 dist-test-slave-5309 krb5kdc[19598](info): set up 2 sockets
Sep 29 00:50:23 dist-test-slave-5309 krb5kdc[19598](info): commencing operation
krb5kdc: starting...
W20260929 00:50:24.431031 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:24 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:50:24.556770 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43329
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:24.675359 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:24.675705 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:24.675789 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:24.679848 19611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:24.679970 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:24.680009 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:24.680071 19611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:24.680109 19611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:24.686167 19611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:24.687345 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:24.688421 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:24.697450 19616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:24.697633 19619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:24.697669 19611 server_base.cc:1034] running on GCE node
W20260929 00:50:24.697427 19617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:24.698485 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:24.699683 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:24.700896 19611 hybrid_clock.cc:648] HybridClock initialized: now 1790643024700875 us; error 33 us; skew 500 ppm
Sep 29 00:50:24 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643024, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:24.703749 19611 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:24.704650 19611 webserver.cc:460] Webserver started at http://127.11.40.254:34677/ using document root <none> and password file <none>
I20260929 00:50:24.705258 19611 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:24.705322 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:24.705498 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:24.707609 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "cadb63bbd1a9408aa1d98dcb75276c72"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "d07983f988cc5544737a76be62259452"
server_key_iv: "6990ab42604b33b6bea85a3ce991657a"
server_key_version: "encryptionkey@0"
I20260929 00:50:24.708125 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "cadb63bbd1a9408aa1d98dcb75276c72"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "d07983f988cc5544737a76be62259452"
server_key_iv: "6990ab42604b33b6bea85a3ce991657a"
server_key_version: "encryptionkey@0"
I20260929 00:50:24.711970 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:24.714740 19626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:24.716418 19611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:24.716557 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "cadb63bbd1a9408aa1d98dcb75276c72"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "d07983f988cc5544737a76be62259452"
server_key_iv: "6990ab42604b33b6bea85a3ce991657a"
server_key_version: "encryptionkey@0"
I20260929 00:50:24.716660 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:24.732511 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:24.735666 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:24.735826 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:24.743191 19611 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:43329
I20260929 00:50:24.743216 19678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:43329 every 8 connection(s)
I20260929 00:50:24.744153 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:24.747014 19679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:24.752560 19679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Bootstrap starting.
I20260929 00:50:24.754127 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19611
I20260929 00:50:24.754280 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:24.754583 11427 external_mini_cluster.cc:1428] Setting key fa53a9d3a2e67f6e59505c94480fbe78
I20260929 00:50:24.755218 19679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:24.756165 19679 log.cc:824] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:24.758368 19679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: No bootstrap required, opened a new log
I20260929 00:50:24.762815 19679 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:24.763111 19679 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:24.763180 19679 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadb63bbd1a9408aa1d98dcb75276c72, State: Initialized, Role: FOLLOWER
Sep 29 00:50:24 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:24.763700 19679 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:24.763873 19679 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:24.763938 19679 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:24.764035 19679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:24.765045 19679 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:24.765394 19679 leader_election.cc:302] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadb63bbd1a9408aa1d98dcb75276c72; no voters:
I20260929 00:50:24.765789 19679 leader_election.cc:288] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:24.766000 19684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:24.766288 19684 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 LEADER]: Becoming Leader. State: Replica: cadb63bbd1a9408aa1d98dcb75276c72, State: Running, Role: LEADER
I20260929 00:50:24.766656 19684 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:24.767165 19679 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:24.768450 19686 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadb63bbd1a9408aa1d98dcb75276c72. Latest consensus state: current_term: 1 leader_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } } }
I20260929 00:50:24.768573 19686 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:24.769042 19685 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } } }
I20260929 00:50:24.769140 19685 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:24.770675 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:24.755758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.11.40.254:43329)
0929 00:50:24.756209 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:24.756227 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:24.756261 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:50:24.757000 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:24.757020 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:24.757342 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:24.757617 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:24.758544 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:24.759511 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:24.760712 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:24.761238 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:24.761356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:24.764459 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:24.764464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:24.764477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:24.764505 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:24.766682 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:24.767193 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:24.767199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:24.767205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:24.767349 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:24.767743 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:24.767747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:24.767749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:24.768320 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:24.769206 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:24.769372 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 00:50:24.769673 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":146,"threads_started":1}
W20260929 00:50:24.772115 19699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:50:24.772207 19699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:50:24.772306 19694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:24.773144 19694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:24.778725 19694 catalog_manager.cc:1347] Generated new cluster ID: 43daf4c6c3f84e69b8d8ee9acff12054
I20260929 00:50:24.778796 19694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:24.797094 19694 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:24.798401 19694 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:24.817281 19694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Generated new TSK 0
I20260929 00:50:24.818231 19694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:24.849099 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:24.963217 19705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:24.963536 19705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:24.963610 19705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:24.968231 19705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:24.968333 19705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:24.968437 19705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:24.973984 19705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:24.975210 19705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:24.976300 19705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:24.986306 19713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:24.986263 19711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:24.986281 19705 server_base.cc:1034] running on GCE node
W20260929 00:50:24.986284 19710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:24.986941 19705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:24.987705 19705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:24.988886 19705 hybrid_clock.cc:648] HybridClock initialized: now 1790643024988871 us; error 34 us; skew 500 ppm
Sep 29 00:50:24 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643024, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:24.991730 19705 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:24.992692 19705 webserver.cc:460] Webserver started at http://127.11.40.193:46059/ using document root <none> and password file <none>
I20260929 00:50:24.993300 19705 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:24.993363 19705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:24.993587 19705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:24.995806 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "6faee6c3d08e4abd9426f3e4ca50e09c"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "4d5232d569eb69ca84893d90584167b7"
server_key_iv: "16d31446cfca03900eb3950ba0880969"
server_key_version: "encryptionkey@0"
I20260929 00:50:24.996380 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "6faee6c3d08e4abd9426f3e4ca50e09c"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "4d5232d569eb69ca84893d90584167b7"
server_key_iv: "16d31446cfca03900eb3950ba0880969"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.000190 19705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:25.002767 19720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.004066 19705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:50:25.004216 19705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "6faee6c3d08e4abd9426f3e4ca50e09c"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "4d5232d569eb69ca84893d90584167b7"
server_key_iv: "16d31446cfca03900eb3950ba0880969"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.004336 19705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:25.015021 19705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:25.018064 19705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:25.018275 19705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:25.018877 19705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:25.019737 19705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:25.019806 19705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.019872 19705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:25.019904 19705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.029225 19705 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43191
I20260929 00:50:25.029295 19833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43191 every 8 connection(s)
I20260929 00:50:25.030151 19705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:25.036145 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19705
I20260929 00:50:25.036263 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:25.036635 11427 external_mini_cluster.cc:1428] Setting key 677818ff43c143e0aea317ba726b4d9d
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643024, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:25.045430 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.032018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:35011 (local address 127.11.40.254:43329)
0929 00:50:25.032166 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:25.032169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:25.033065 (+ 896us) server_negotiation.cc:407] Connection header received
0929 00:50:25.034308 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:25.034310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:25.034361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:25.034473 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:25.035802 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.036476 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.037998 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.038271 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.038390 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.041190 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:25.041196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:25.041202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:25.041237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:25.043038 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.043625 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.043631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.043635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.043741 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.044105 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.044111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.044116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.044389 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:25.044507 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:25.045089 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 00:50:25.045227 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:50:25.046224 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.032400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:35011)
0929 00:50:25.032904 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:25.032952 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:25.034106 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:25.034613 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:25.034620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:25.035023 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:25.035661 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.035678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.036611 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.036615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.037862 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.037873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.038470 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.038478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.038639 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.039296 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:25.039313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:25.041013 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:25.043173 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.043178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.043192 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.043496 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.043842 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.043844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.043845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.043993 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.044500 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:25.044505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:25.044751 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 00:50:25.044987 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:50:25.045269 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:25.047616 19834 heartbeater.cc:347] Connected to a master server at 127.11.40.254:43329
I20260929 00:50:25.047833 19834 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:25.048317 19834 heartbeater.cc:511] Master 127.11.40.254:43329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:50:25.050007 19643 ts_manager.cc:194] Registered new tserver with Master: 6faee6c3d08e4abd9426f3e4ca50e09c (127.11.40.193:43191)
I20260929 00:50:25.051692 19643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:35011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:25.068965 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:25.181406 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:25.181720 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:25.181793 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:25.186051 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:25.186134 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:25.186220 19839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:25.191469 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:25.192589 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:25.193702 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:25.203678 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:25.203765 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.203815 19839 server_base.cc:1034] running on GCE node
W20260929 00:50:25.203765 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.204475 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:25.205296 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:25.206483 19839 hybrid_clock.cc:648] HybridClock initialized: now 1790643025206459 us; error 35 us; skew 500 ppm
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:25.209483 19839 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:25.210381 19839 webserver.cc:460] Webserver started at http://127.11.40.194:36917/ using document root <none> and password file <none>
I20260929 00:50:25.211014 19839 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:25.211081 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:25.211313 19839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:25.213510 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "b192ed79f1ff437db5e993ac8787bef4"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "2dca3cecf6a37ceab66ae69d30591419"
server_key_iv: "9db49487a72dfee23c2bc5517a164526"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.214143 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "b192ed79f1ff437db5e993ac8787bef4"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "2dca3cecf6a37ceab66ae69d30591419"
server_key_iv: "9db49487a72dfee23c2bc5517a164526"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.218016 19839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 00:50:25.220533 19854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.222086 19839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:25.222239 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "b192ed79f1ff437db5e993ac8787bef4"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "2dca3cecf6a37ceab66ae69d30591419"
server_key_iv: "9db49487a72dfee23c2bc5517a164526"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.222366 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:25.241501 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:25.245117 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:25.245342 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:25.245985 19839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:25.246846 19839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:25.246896 19839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.246975 19839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:25.247030 19839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.256676 19839 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:38763
I20260929 00:50:25.256698 19967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:38763 every 8 connection(s)
I20260929 00:50:25.257651 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:25.265898 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19839
I20260929 00:50:25.266072 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:25.266413 11427 external_mini_cluster.cc:1428] Setting key 07e016c6dc8956c09c40ccb71a733e33
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:25.272395 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.259577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:50459 (local address 127.11.40.254:43329)
0929 00:50:25.259740 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:25.259744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:25.260629 (+ 885us) server_negotiation.cc:407] Connection header received
0929 00:50:25.261956 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:25.261959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:25.262017 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:25.262130 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:25.263360 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.263895 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.265118 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.265351 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.265491 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.268471 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:25.268477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:25.268485 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:25.268545 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:25.270259 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.270815 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.270819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.270821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.270917 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.271272 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.271276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.271278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.271476 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:25.271593 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:25.272097 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 00:50:25.272227 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:50:25.273341 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.259994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:50459)
0929 00:50:25.260459 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:25.260504 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:25.261734 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:25.262312 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:25.262319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:25.262706 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:25.263205 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.263219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.264029 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.264032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.264982 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.264991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.265544 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.265548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.265743 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.266427 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:25.266444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:25.268289 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:25.270397 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.270404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.270420 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.270695 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.271014 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.271019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.271021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.271173 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.271582 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:25.271589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:25.271855 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 00:50:25.272059 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 00:50:25.272417 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260929 00:50:25.274972 19968 heartbeater.cc:347] Connected to a master server at 127.11.40.254:43329
I20260929 00:50:25.275262 19968 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:25.275905 19968 heartbeater.cc:511] Master 127.11.40.254:43329 requested a full tablet report, sending...
I20260929 00:50:25.277245 19643 ts_manager.cc:194] Registered new tserver with Master: b192ed79f1ff437db5e993ac8787bef4 (127.11.40.194:38763)
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:25.277989 19643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:50459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:25.299724 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:25.415685 19973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:25.415984 19973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:25.416029 19973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:25.420205 19973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:25.420327 19973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:25.420465 19973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:25.426275 19973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:25.427445 19973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:25.428515 19973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:25.438808 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.438903 19973 server_base.cc:1034] running on GCE node
W20260929 00:50:25.438781 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:25.438778 19981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.439430 19973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:25.440100 19973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:25.441259 19973 hybrid_clock.cc:648] HybridClock initialized: now 1790643025441246 us; error 31 us; skew 500 ppm
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:25.444177 19973 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:25.445250 19973 webserver.cc:460] Webserver started at http://127.11.40.195:34387/ using document root <none> and password file <none>
I20260929 00:50:25.445888 19973 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:25.445986 19973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:25.446235 19973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:25.449034 19973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "b30ae158746f4db58f7a2561c9bcbbe0"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "efd7554b5efee9f0ee005f28f659da55"
server_key_iv: "a2a41c2a5d2e206bf9d39f51d3a9b76b"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.449674 19973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "b30ae158746f4db58f7a2561c9bcbbe0"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "efd7554b5efee9f0ee005f28f659da55"
server_key_iv: "a2a41c2a5d2e206bf9d39f51d3a9b76b"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.453924 19973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:50:25.457021 19988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.458374 19973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:25.458536 19973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "b30ae158746f4db58f7a2561c9bcbbe0"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "efd7554b5efee9f0ee005f28f659da55"
server_key_iv: "a2a41c2a5d2e206bf9d39f51d3a9b76b"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.458667 19973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:25.477872 19973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:25.481644 19973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:25.481874 19973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:25.482538 19973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:25.483407 19973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:25.483480 19973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.483554 19973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:25.483606 19973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.493409 19973 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:40815
I20260929 00:50:25.493438 20101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:40815 every 8 connection(s)
I20260929 00:50:25.494370 19973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:25.497012 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 19973
I20260929 00:50:25.497166 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:25.497493 11427 external_mini_cluster.cc:1428] Setting key c5fd7f6174d4c3dac42a7502dc73f07f
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:25.509433 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.496505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:34583 (local address 127.11.40.254:43329)
0929 00:50:25.496646 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:25.496650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:25.497730 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 00:50:25.499259 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:25.499262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:25.499319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:25.499397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:25.500733 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.501365 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.502544 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.502777 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.502904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.505765 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:25.505768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:25.505771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:25.505798 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:25.507129 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.507735 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.507743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.507745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.507822 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:25.508194 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:25.508197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:25.508199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:25.508397 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:25.508516 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:25.509119 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 00:50:25.509256 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:50:25.510426 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:25.496941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:34583)
0929 00:50:25.497513 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:25.497596 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:25.499047 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:25.499577 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:25.499585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:25.500046 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:25.500571 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.500588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.501500 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.501504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.502408 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:25.502418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:25.502945 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:25.502949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:25.503126 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:25.503716 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:25.503732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:25.505550 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:25.507275 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.507281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.507299 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.507590 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.507937 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:25.507940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:25.507942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:25.508087 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:25.508514 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:25.508523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:25.508795 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 00:50:25.509078 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 00:50:25.509436 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260929 00:50:25.511687 20102 heartbeater.cc:347] Connected to a master server at 127.11.40.254:43329
I20260929 00:50:25.511965 20102 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:25.512496 20102 heartbeater.cc:511] Master 127.11.40.254:43329 requested a full tablet report, sending...
I20260929 00:50:25.513645 19643 ts_manager.cc:194] Registered new tserver with Master: b30ae158746f4db58f7a2561c9bcbbe0 (127.11.40.195:40815)
I20260929 00:50:25.514242 19643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:34583
I20260929 00:50:25.523885 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
Entry for principal oryx/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab.
WARNING: no policy specified for oryx/127.11.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
Entry for principal oryx/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab.
WARNING: no policy specified for oryx/127.11.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
Entry for principal oryx/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab.
I20260929 00:50:25.615804 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19705
I20260929 00:50:25.621696 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19839
I20260929 00:50:25.627513 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19973
I20260929 00:50:25.633119 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 19611
I20260929 00:50:25.639417 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43329
--webserver_interface=127.11.40.254
--webserver_port=34677
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:25.753854 20111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:25.754155 20111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:25.754197 20111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:25.758374 20111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:25.758455 20111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:25.758476 20111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:25.758493 20111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:25.758508 20111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:25.763965 20111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:25.765168 20111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:25.766263 20111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:25.775128 20116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:25.775163 20119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:25.775163 20117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.775476 20111 server_base.cc:1034] running on GCE node
I20260929 00:50:25.776065 20111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:25.777285 20111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:25.778489 20111 hybrid_clock.cc:648] HybridClock initialized: now 1790643025778474 us; error 30 us; skew 500 ppm
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:25.781366 20111 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:25.782320 20111 webserver.cc:460] Webserver started at http://127.11.40.254:34677/ using document root <none> and password file <none>
I20260929 00:50:25.782892 20111 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:25.783018 20111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:25.787107 20111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:25.789911 20126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.791671 20111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:25.791812 20111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "cadb63bbd1a9408aa1d98dcb75276c72"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "d07983f988cc5544737a76be62259452"
server_key_iv: "6990ab42604b33b6bea85a3ce991657a"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.792598 20111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:25.811558 20111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:25.814715 20111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:25.814939 20111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:25.822228 20111 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:43329
I20260929 00:50:25.822247 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:43329 every 8 connection(s)
I20260929 00:50:25.823346 20111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:25.826577 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20111
I20260929 00:50:25.827068 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:43191
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:43329
--webserver_port=46059
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:50:25.828323 20179 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:50:25.830555 20179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Bootstrap starting.
I20260929 00:50:25.847966 20179 log.cc:824] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:25.855255 20179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:50:25.855716 20179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Bootstrap complete.
I20260929 00:50:25.860270 20179 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:25.860714 20179 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadb63bbd1a9408aa1d98dcb75276c72, State: Initialized, Role: FOLLOWER
I20260929 00:50:25.861275 20179 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:25.861403 20179 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:25.861502 20179 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:25.861600 20179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:50:25.863691 20179 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:25.864048 20179 leader_election.cc:302] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadb63bbd1a9408aa1d98dcb75276c72; no voters:
I20260929 00:50:25.864439 20179 leader_election.cc:288] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:50:25.864532 20183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:50:25.864722 20183 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [term 2 LEADER]: Becoming Leader. State: Replica: cadb63bbd1a9408aa1d98dcb75276c72, State: Running, Role: LEADER
I20260929 00:50:25.865059 20183 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } }
I20260929 00:50:25.865757 20179 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:25.866468 20184 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } } }
I20260929 00:50:25.866598 20184 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:25.867270 20190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:25.867301 20185 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadb63bbd1a9408aa1d98dcb75276c72. Latest consensus state: current_term: 2 leader_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadb63bbd1a9408aa1d98dcb75276c72" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43329 } } }
I20260929 00:50:25.867422 20185 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:25.870718 20190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:25.871490 20190 catalog_manager.cc:1259] Loaded cluster ID: 43daf4c6c3f84e69b8d8ee9acff12054
I20260929 00:50:25.871551 20190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:25.874091 20190 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:25.874862 20190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cadb63bbd1a9408aa1d98dcb75276c72: Loaded TSK: 0
I20260929 00:50:25.876106 20190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:50:25.946889 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:25.947178 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:25.947224 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:25.951483 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:25.951565 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:25.951627 20181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:25.957248 20181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.193.keytab
--principal=oryx/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:43191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=46059
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:25.958328 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:25.959358 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:25.970897 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.971076 20181 server_base.cc:1034] running on GCE node
W20260929 00:50:25.971174 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:25.971463 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:25.971923 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:25.972581 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:25.973819 20181 hybrid_clock.cc:648] HybridClock initialized: now 1790643025973789 us; error 63 us; skew 500 ppm
Sep 29 00:50:25 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643025, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:25.976946 20181 init.cc:377] Logged in from keytab as oryx/127.11.40.193@KRBTEST.COM (short username oryx)
I20260929 00:50:25.977916 20181 webserver.cc:460] Webserver started at http://127.11.40.193:46059/ using document root <none> and password file <none>
I20260929 00:50:25.978500 20181 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:25.978605 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:25.982827 20181 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:25.985204 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:25.986573 20181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:25.986708 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "6faee6c3d08e4abd9426f3e4ca50e09c"
format_stamp: "Formatted at 2026-09-29 00:50:24 on dist-test-slave-5309"
server_key: "4d5232d569eb69ca84893d90584167b7"
server_key_iv: "16d31446cfca03900eb3950ba0880969"
server_key_version: "encryptionkey@0"
I20260929 00:50:25.987278 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:26.001073 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:26.004534 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:26.004772 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:26.005453 20181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:26.006362 20181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:26.006433 20181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.006508 20181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:26.006553 20181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.017316 20181 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43191
I20260929 00:50:26.017381 20328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43191 every 8 connection(s)
I20260929 00:50:26.018407 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:26.024693 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20181
I20260929 00:50:26.025242 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:38763
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:43329
--webserver_port=36917
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:50:26.033455 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.020507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:35277 (local address 127.11.40.254:43329)
0929 00:50:26.020992 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.021015 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.021336 (+ 321us) server_negotiation.cc:407] Connection header received
0929 00:50:26.022858 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.022870 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.023185 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.023494 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.024631 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.026302 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.027633 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.028191 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.028403 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.032626 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:35277: BlockingRecv error: recv got EOF from 127.11.40.193:35277 (error 108)
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260929 00:50:26.033581 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.020665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:35277)
0929 00:50:26.021165 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.021218 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.022434 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.023580 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.023587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.023960 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.024433 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.024447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.026489 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.026492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.027459 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.027471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.028397 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.028401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.028603 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.029407 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.029425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.032408 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":186,"threads_started":1}
W20260929 00:50:26.034108 20329 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:43329 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.40.254:43329: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.040750 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.034655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:37235)
0929 00:50:26.034863 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.034881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.034991 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.035255 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.035258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.035474 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.035636 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.035647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.036438 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.036444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.037494 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.037506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.037948 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.037952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.038049 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.038533 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.038548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.040610 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 00:50:26.040934 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.034735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37235 (local address 127.11.40.254:43329)
0929 00:50:26.034876 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.034880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.034947 (+ 67us) server_negotiation.cc:407] Connection header received
0929 00:50:26.035089 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.035091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.035133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.035205 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.035793 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.036312 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.037620 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.037840 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.037922 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.040756 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:37235: BlockingRecv error: recv got EOF from 127.11.40.193:37235 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.047142 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.041486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:59273)
0929 00:50:26.041833 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.041846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.041969 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.042264 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.042267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.042474 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.042744 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.042751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.043548 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.043553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.044082 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.044090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.044555 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.044558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.044646 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.045172 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.045184 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.046985 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257}
W20260929 00:50:26.047228 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.041633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59273 (local address 127.11.40.254:43329)
0929 00:50:26.041775 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.041779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.041920 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:50:26.042083 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.042087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.042142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.042249 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.042890 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.043432 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.044215 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.044431 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.044539 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.047113 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:59273: BlockingRecv error: recv got EOF from 127.11.40.193:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 00:50:26.047585 20329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:50:26.145036 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:26.145361 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:26.145438 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:26.149595 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:26.149689 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:26.149783 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:26.155125 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.194.keytab
--principal=oryx/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:38763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=36917
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:26.156252 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:26.157449 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:26.167694 20339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:26.167785 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:26.167759 20333 server_base.cc:1034] running on GCE node
W20260929 00:50:26.167703 20341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:26.168459 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:26.169160 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:26.170325 20333 hybrid_clock.cc:648] HybridClock initialized: now 1790643026170282 us; error 58 us; skew 500 ppm
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643026, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:26.173099 20333 init.cc:377] Logged in from keytab as oryx/127.11.40.194@KRBTEST.COM (short username oryx)
I20260929 00:50:26.174090 20333 webserver.cc:460] Webserver started at http://127.11.40.194:36917/ using document root <none> and password file <none>
I20260929 00:50:26.174703 20333 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:26.174779 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:26.178771 20333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:50:26.180909 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.182124 20333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:26.182246 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "b192ed79f1ff437db5e993ac8787bef4"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "2dca3cecf6a37ceab66ae69d30591419"
server_key_iv: "9db49487a72dfee23c2bc5517a164526"
server_key_version: "encryptionkey@0"
I20260929 00:50:26.182824 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:26.201740 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:26.204865 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:26.205097 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:26.205775 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:26.206620 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:26.206689 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.206756 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:26.206804 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.216565 20333 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:38763
I20260929 00:50:26.216614 20461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:38763 every 8 connection(s)
I20260929 00:50:26.217675 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:26.222715 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20333
I20260929 00:50:26.223196 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:40815
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:43329
--webserver_port=34387
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:50:26.230392 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.219653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:60895 (local address 127.11.40.254:43329)
0929 00:50:26.219803 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.219807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.220720 (+ 913us) server_negotiation.cc:407] Connection header received
0929 00:50:26.222285 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.222288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.222362 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.222476 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.224071 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.224663 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.225976 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.226232 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.226364 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.230226 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:60895: BlockingRecv error: recv got EOF from 127.11.40.194:60895 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:50:26.231106 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.220079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:60895)
0929 00:50:26.220551 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.220598 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.222030 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.222678 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.222687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.223199 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.223897 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.223914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.224842 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.224845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.225828 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.225838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.226422 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.226427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.226614 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.227308 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.227326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.230072 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
W20260929 00:50:26.231666 20462 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:43329 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.40.254:43329: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.238428 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.232214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:38651)
0929 00:50:26.232437 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.232456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.232566 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.232881 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.232886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.233087 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.233240 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.233251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.234054 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.234058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.234973 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.234982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.235434 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.235438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.235535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.236044 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.236058 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.238284 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260929 00:50:26.238852 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.232313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:38651 (local address 127.11.40.254:43329)
0929 00:50:26.232442 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.232446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.232507 (+ 61us) server_negotiation.cc:407] Connection header received
0929 00:50:26.232674 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.232676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.232717 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.232792 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.233383 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.233942 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.235096 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.235323 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.235403 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.238466 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:38651: BlockingRecv error: recv got EOF from 127.11.40.194:38651 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.244972 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.238967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:34841)
0929 00:50:26.239210 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.239224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.239294 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.239620 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.239623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.239787 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.239879 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.239885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.240656 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.240662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.241584 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.241594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.242048 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.242052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.242133 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.242657 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.242672 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.244781 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175}
W20260929 00:50:26.245222 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.239211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:34841 (local address 127.11.40.254:43329)
0929 00:50:26.239427 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.239430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.239443 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:26.239476 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.239478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.239514 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.239582 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.240013 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.240541 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.241709 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.241938 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.242022 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.244974 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:34841: BlockingRecv error: recv got EOF from 127.11.40.194:34841 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
W20260929 00:50:26.245237 20462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:50:26.343783 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:26.344090 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:26.344136 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:26.348191 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:26.348263 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:26.348323 20465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:26.353904 20465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/krb5kdc/oryx_127.11.40.195.keytab
--principal=oryx/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=34387
--tserver_master_addrs=127.11.40.254:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:26.354980 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:26.355978 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:26.365968 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:26.365968 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:26.366458 20465 server_base.cc:1034] running on GCE node
W20260929 00:50:26.365998 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:26.367098 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:26.367970 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:26.369181 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790643026369159 us; error 39 us; skew 500 ppm
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643026, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:26.373006 20465 init.cc:377] Logged in from keytab as oryx/127.11.40.195@KRBTEST.COM (short username oryx)
I20260929 00:50:26.374032 20465 webserver.cc:460] Webserver started at http://127.11.40.195:34387/ using document root <none> and password file <none>
I20260929 00:50:26.374861 20465 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:26.374948 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:26.379420 20465 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 00:50:26.381762 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.383198 20465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:26.383304 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "b30ae158746f4db58f7a2561c9bcbbe0"
format_stamp: "Formatted at 2026-09-29 00:50:25 on dist-test-slave-5309"
server_key: "efd7554b5efee9f0ee005f28f659da55"
server_key_iv: "a2a41c2a5d2e206bf9d39f51d3a9b76b"
server_key_version: "encryptionkey@0"
I20260929 00:50:26.383900 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:26.412559 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:26.416227 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:26.416460 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:26.417171 20465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:26.418036 20465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:26.418084 20465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.418150 20465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:26.418185 20465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:26.427819 20465 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:40815
I20260929 00:50:26.427843 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:40815 every 8 connection(s)
I20260929 00:50:26.428912 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:26.430472 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20465
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:50:26.440611 20331 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.431031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:51067 (local address 127.11.40.254:43329)
0929 00:50:26.431190 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.431195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.432276 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 00:50:26.433847 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.433850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.433902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.433976 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.435172 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.435673 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.436856 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.437075 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.437189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.440487 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:51067: BlockingRecv error: recv got EOF from 127.11.40.195:51067 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:50:26.440853 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.431601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.11.40.254:43329)
0929 00:50:26.431860 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.431873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.431889 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:26.431941 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.431943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.431991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.432094 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.433152 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.433731 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.434636 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.434887 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.435006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.436500 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:26.436506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:26.436520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:26.436549 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:26.438476 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:26.438947 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:26.438974 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:26.438983 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:26.439125 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:26.439425 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:26.439430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:26.439432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:26.439844 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:26.440064 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:26.440323 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 00:50:26.440588 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260929 00:50:26.441376 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.431570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:51067)
0929 00:50:26.432105 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.432161 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.433626 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.434169 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.434176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.434536 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.435032 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.435045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.435805 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.435814 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.436703 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.436712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.437229 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.437233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.437406 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.437984 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.438004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.440376 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":153,"threads_started":1}
W20260929 00:50:26.441879 20594 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:43329 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.40.254:43329: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.448303 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.442347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:58953)
0929 00:50:26.442621 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.442639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.442750 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.442997 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.443000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.443170 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.443261 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.443266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.444227 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.444231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.445004 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.445015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.445548 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.445551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.445638 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.446165 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.446177 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.448190 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204}
W20260929 00:50:26.448508 20597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.442392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:58953 (local address 127.11.40.254:43329)
0929 00:50:26.442527 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.442533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.442705 (+ 172us) server_negotiation.cc:407] Connection header received
0929 00:50:26.442842 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.442850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.442887 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.442982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.443434 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.444075 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.445169 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.445412 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.445502 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.448345 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:58953: BlockingRecv error: recv got EOF from 127.11.40.195:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:26 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:26.453493 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:26.449015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:48015)
0929 00:50:26.449140 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:26.449157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:26.449240 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:26.449456 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:26.449458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:26.449580 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:26.449650 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.449654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.450304 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.450306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.450813 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:26.450820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.451198 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.451200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:26.451254 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.451673 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:26.451684 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:26.453395 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 00:50:26.453666 20597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:26.449091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:48015 (local address 127.11.40.254:43329)
0929 00:50:26.449245 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:26.449248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:26.449261 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:26.449298 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:26.449300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:26.449334 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:26.449429 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:26.449741 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.450219 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.450908 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:26.451106 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:26.451179 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:26.453510 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:48015: BlockingRecv error: recv got EOF from 127.11.40.195:48015 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 00:50:26.453784 20594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:27.054627 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:27.048315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:55531)
0929 00:50:27.048686 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:27.048701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:27.048876 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:27.049176 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:27.049178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:27.049361 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:27.049481 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.049486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.050360 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.050363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.050942 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.050951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.051442 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.051444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.051522 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.052335 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:27.052347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:27.054445 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":181,"threads_started":1}
W20260929 00:50:27.054672 20600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:27.048498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:55531 (local address 127.11.40.254:43329)
0929 00:50:27.048786 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:27.048790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:27.048830 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:50:27.049004 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:27.049006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:27.049053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:27.049139 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:27.049595 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.050187 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.051054 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.051322 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.051411 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.054542 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:55531: BlockingRecv error: recv got EOF from 127.11.40.193:55531 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:27.252068 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:27.246074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:59413)
0929 00:50:27.246441 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:27.246456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:27.246602 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:27.246966 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:27.246969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:27.247238 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:27.247363 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.247370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.248155 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.248158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.248747 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.248756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.249304 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.249308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.249380 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.250011 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:27.250032 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:27.251936 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
W20260929 00:50:27.252295 20600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:27.246123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:59413 (local address 127.11.40.254:43329)
0929 00:50:27.246248 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:27.246252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:27.246559 (+ 307us) server_negotiation.cc:407] Connection header received
0929 00:50:27.246729 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:27.246734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:27.246825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:27.246916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:27.247498 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.248030 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.248964 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.249191 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.249277 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.252129 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:59413: BlockingRecv error: recv got EOF from 127.11.40.194:59413 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:27 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:27.460793 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:27.454847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:46445)
0929 00:50:27.455222 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:27.455238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:27.455359 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:27.455601 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:27.455604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:27.455803 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:27.455916 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.455922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.456760 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.456763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.457518 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:27.457530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.458052 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.458056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:27.458128 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.458749 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:27.458762 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:27.460658 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":161,"threads_started":1}
W20260929 00:50:27.461293 20600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:27.454938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:46445 (local address 127.11.40.254:43329)
0929 00:50:27.455214 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:27.455218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:27.455347 (+ 129us) server_negotiation.cc:407] Connection header received
0929 00:50:27.455434 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:27.455436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:27.455485 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:27.455555 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:27.456032 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.456613 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.457667 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:27.457903 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:27.457994 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:27.461080 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:46445: BlockingRecv error: recv got EOF from 127.11.40.195:46445 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:28.062009 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:28.055733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:60857)
0929 00:50:28.056067 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:28.056084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:28.056226 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:28.056576 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:28.056578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:28.056762 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:28.056949 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.056956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.057752 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.057755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.058344 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.058353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.058802 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.058804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.058892 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.059398 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:28.059410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:28.061568 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":80,"threads_started":1}
W20260929 00:50:28.062091 20604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:28.055797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:60857 (local address 127.11.40.254:43329)
0929 00:50:28.056088 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:28.056117 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:28.056141 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:50:28.056387 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:28.056389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:28.056442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:28.056527 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:28.057066 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.057615 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.058468 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.058697 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.058779 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.061953 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:60857: BlockingRecv error: recv got EOF from 127.11.40.193:60857 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:28.258615 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:28.253078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:43523)
0929 00:50:28.253365 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:28.253379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:28.253475 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:28.253719 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:28.253721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:28.253910 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:28.254007 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.254013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.254824 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.254827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.255436 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.255445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.255929 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.255931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.255998 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.256572 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:28.256583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:28.258478 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
W20260929 00:50:28.258975 20604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:28.253219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:43523 (local address 127.11.40.254:43329)
0929 00:50:28.253392 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:28.253396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:28.253433 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:50:28.253563 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:28.253565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:28.253610 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:28.253683 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:28.254143 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.254687 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.255564 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.255807 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.255893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.258672 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:43523: BlockingRecv error: recv got EOF from 127.11.40.194:43523 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:28 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:28.472879 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:28.461890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:52677)
0929 00:50:28.462164 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:28.462178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:28.462268 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:28.462573 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:28.462575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:28.462768 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:28.462867 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.462874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.463613 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.463615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.464159 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:28.464167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.464675 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.464678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:28.464742 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.470789 (+ 6047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:28.470803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:28.472674 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
W20260929 00:50:28.473140 20604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:28.462053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52677 (local address 127.11.40.254:43329)
0929 00:50:28.462211 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:28.462214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:28.462228 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:28.462391 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:28.462393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:28.462439 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:28.462509 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:28.462973 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.463520 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.464316 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:28.464566 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:28.464651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:28.472961 (+ 8310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:52677: BlockingRecv error: recv got EOF from 127.11.40.195:52677 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:29.068480 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:29.062876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:37291)
0929 00:50:29.063247 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:29.063261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:29.063353 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:29.063618 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:29.063620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:29.063803 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:29.063919 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.063925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.064659 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.064662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.065276 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.065285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.065792 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.065795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.065860 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.066351 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:29.066362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:29.068338 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
W20260929 00:50:29.068651 20608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:29.063041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37291 (local address 127.11.40.254:43329)
0929 00:50:29.063309 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:29.063313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:29.063326 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:29.063465 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:29.063467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:29.063509 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:29.063584 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:29.064022 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.064564 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.065394 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.065692 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.065774 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.068499 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:37291: BlockingRecv error: recv got EOF from 127.11.40.193:37291 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:29.265039 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:29.259444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:52153)
0929 00:50:29.259712 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:29.259738 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:29.259861 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:29.260111 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:29.260113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:29.260381 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:29.260486 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.260492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.261352 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.261355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.261919 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.261928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.262368 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.262370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.262432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.262938 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:29.262952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:29.264907 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
W20260929 00:50:29.265223 20608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:29.259515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:52153 (local address 127.11.40.254:43329)
0929 00:50:29.259669 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:29.259673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:29.259823 (+ 150us) server_negotiation.cc:407] Connection header received
0929 00:50:29.259951 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:29.259954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:29.259998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:29.260069 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:29.260602 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.261235 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.262040 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.262253 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.262329 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.265095 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:52153: BlockingRecv error: recv got EOF from 127.11.40.194:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:29 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:29.479450 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:29.473765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:44577)
0929 00:50:29.474050 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:29.474065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:29.474178 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:29.474456 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:29.474458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:29.474636 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:29.474737 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.474742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.475576 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.475579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.476149 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:29.476157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.476647 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.476650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:29.476735 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.477354 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:29.477369 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:29.479299 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260929 00:50:29.479609 20608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:29.473842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:44577 (local address 127.11.40.254:43329)
0929 00:50:29.473966 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:29.473970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:29.474164 (+ 194us) server_negotiation.cc:407] Connection header received
0929 00:50:29.474289 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:29.474291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:29.474344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:29.474420 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:29.474900 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.475460 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.476253 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:29.476488 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:29.476568 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:29.479480 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:44577: BlockingRecv error: recv got EOF from 127.11.40.195:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:30.074918 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:30.069318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:56457)
0929 00:50:30.069617 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:30.069631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:30.069723 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:30.070045 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:30.070047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:30.070236 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:30.070338 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.070343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.071095 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.071097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.071656 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.071664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.072150 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.072155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.072236 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.072903 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:30.072915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:30.074769 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
W20260929 00:50:30.075258 20612 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:30.069562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:56457 (local address 127.11.40.254:43329)
0929 00:50:30.069796 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:30.069799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:30.069812 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:30.069869 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:30.069871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:30.069914 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:30.070008 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:30.070446 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.070991 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.071768 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.072010 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.072089 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.075005 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:56457: BlockingRecv error: recv got EOF from 127.11.40.193:56457 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:30.272130 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:30.266191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:56077)
0929 00:50:30.266516 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:30.266531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:30.266718 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:30.267242 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:30.267245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:30.267440 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:30.267540 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.267545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.268404 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.268407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.269041 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.269051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.269487 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.269490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.269559 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.270079 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:30.270090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:30.271952 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
W20260929 00:50:30.272235 20612 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:30.266360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56077 (local address 127.11.40.254:43329)
0929 00:50:30.266530 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:30.266540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:30.266666 (+ 126us) server_negotiation.cc:407] Connection header received
0929 00:50:30.266931 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:30.266934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:30.266988 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:30.267154 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:30.267670 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.268289 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.269142 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.269354 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.269437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.272102 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:56077: BlockingRecv error: recv got EOF from 127.11.40.194:56077 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:30 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:30.487083 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:30.480464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:44021)
0929 00:50:30.480786 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:30.480855 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:30.480969 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:30.481355 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:30.481358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:30.481548 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:30.481645 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.481651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.482564 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.482567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.483305 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:30.483317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.483888 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.483891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:30.483961 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.484682 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:30.484695 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:30.486900 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":137,"threads_started":1}
W20260929 00:50:30.487329 20612 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:30.480648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:44021 (local address 127.11.40.254:43329)
0929 00:50:30.480791 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:30.480794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:30.480981 (+ 187us) server_negotiation.cc:407] Connection header received
0929 00:50:30.481042 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:30.481044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:30.481095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:30.481271 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:30.481791 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.482428 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.483476 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:30.483701 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:30.483800 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:30.487119 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:44021: BlockingRecv error: recv got EOF from 127.11.40.195:44021 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:31.081445 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:31.075805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:51201)
0929 00:50:31.076147 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:31.076160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:31.076254 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:31.076515 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:31.076517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:31.076708 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:31.076835 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.076841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.077632 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.077635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.078220 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.078229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.078774 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.078776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.078840 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.079352 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:31.079363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:31.081292 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":152,"threads_started":1}
W20260929 00:50:31.081643 20616 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:31.075890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:51201 (local address 127.11.40.254:43329)
0929 00:50:31.076153 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:31.076157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:31.076237 (+ 80us) server_negotiation.cc:407] Connection header received
0929 00:50:31.076338 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:31.076341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:31.076385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:31.076476 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:31.076932 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.077516 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.078426 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.078661 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.078753 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.081489 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:51201: BlockingRecv error: recv got EOF from 127.11.40.193:51201 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:31.278996 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:31.273158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:47677)
0929 00:50:31.273513 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:31.273531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:31.273656 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:31.273964 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:31.273967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:31.274192 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:31.274287 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.274293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.275090 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.275096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.275675 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.275683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.276154 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.276157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.276242 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.276754 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:31.276766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:31.278813 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
W20260929 00:50:31.279035 20616 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:31.273371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:47677 (local address 127.11.40.254:43329)
0929 00:50:31.273526 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:31.273532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:31.273614 (+ 82us) server_negotiation.cc:407] Connection header received
0929 00:50:31.273776 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:31.273779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:31.273836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:31.273938 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:31.274442 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.274977 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.275797 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.276028 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.276138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.278938 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:47677: BlockingRecv error: recv got EOF from 127.11.40.194:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:31 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:31.494264 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:31.488032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:60467)
0929 00:50:31.488326 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:31.488339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:31.488458 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:31.488763 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:31.488766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:31.489007 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:31.489114 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.489121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.490006 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.490009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.490680 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:31.490690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.491229 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.491232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:31.491350 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.492014 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:31.492028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:31.494130 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
W20260929 00:50:31.494560 20616 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:31.488145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:60467 (local address 127.11.40.254:43329)
0929 00:50:31.488363 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:31.488368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:31.488391 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:50:31.488561 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:31.488563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:31.488609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:31.488678 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:31.489270 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.489881 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.490831 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:31.491057 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:31.491153 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:31.494421 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:60467: BlockingRecv error: recv got EOF from 127.11.40.195:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:32.094040 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:32.086756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:36367)
0929 00:50:32.087037 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:32.087055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:32.087410 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:32.087885 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:32.087888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:32.088189 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:32.088326 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.088333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.089301 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.089307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.089881 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.089890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.090524 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.090527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.090616 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.091613 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:32.091626 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:32.093861 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
W20260929 00:50:32.094162 20620 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:32.087188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:36367 (local address 127.11.40.254:43329)
0929 00:50:32.087558 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:32.087564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:32.087588 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:50:32.087648 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:32.087653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:32.087719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:32.087848 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:32.088470 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.089177 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.090020 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.090377 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.090505 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.094023 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:36367: BlockingRecv error: recv got EOF from 127.11.40.193:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:32.285648 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:32.279823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:34115)
0929 00:50:32.280115 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:32.280131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:32.280273 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:32.280560 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:32.280563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:32.280782 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:32.280943 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.280949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.281718 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.281721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.282290 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.282299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.282786 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.282789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.282879 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.283469 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:32.283490 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:32.285462 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260929 00:50:32.285733 20620 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:32.279915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:34115 (local address 127.11.40.254:43329)
0929 00:50:32.280074 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:32.280079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:32.280218 (+ 139us) server_negotiation.cc:407] Connection header received
0929 00:50:32.280383 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:32.280388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:32.280441 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:32.280537 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:32.281061 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.281597 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.282415 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.282667 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.282780 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.285613 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:34115: BlockingRecv error: recv got EOF from 127.11.40.194:34115 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:32 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:32.501332 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:32.495132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:60655)
0929 00:50:32.495540 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:32.495556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:32.495712 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:32.496025 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:32.496027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:32.496252 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:32.496344 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.496356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.497283 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.497288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.497957 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:32.497972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.498460 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.498463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:32.498551 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.499103 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:32.499114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:32.501155 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
W20260929 00:50:32.501434 20620 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:32.495285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:60655 (local address 127.11.40.254:43329)
0929 00:50:32.495478 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:32.495483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:32.495651 (+ 168us) server_negotiation.cc:407] Connection header received
0929 00:50:32.495836 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:32.495840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:32.495908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:32.495998 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:32.496497 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.497162 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.498104 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:32.498340 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:32.498454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:32.501308 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:60655: BlockingRecv error: recv got EOF from 127.11.40.195:60655 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:33.100937 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:33.095065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:57583)
0929 00:50:33.095407 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:33.095422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:33.095528 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:33.095841 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:33.095843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:33.096032 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:33.096134 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.096139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.097026 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.097029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.097703 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.097713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.098160 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.098162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.098228 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.098786 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:33.098798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:33.100720 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
W20260929 00:50:33.101049 20624 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:33.095163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:57583 (local address 127.11.40.254:43329)
0929 00:50:33.095436 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:33.095440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:33.095477 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:50:33.095618 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:33.095620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:33.095664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:33.095757 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:33.096251 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.096911 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.097827 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.098058 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.098139 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.100893 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:57583: BlockingRecv error: recv got EOF from 127.11.40.193:57583 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:33.293586 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:33.286764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:39739)
0929 00:50:33.287122 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:33.287143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:33.287309 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:33.287637 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:33.287641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:33.287927 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:33.288089 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.288099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.289251 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.289253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.289841 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.289853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.290473 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.290476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.290561 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.291263 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:33.291276 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:33.293458 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
W20260929 00:50:33.293859 20624 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:33.286903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:39739 (local address 127.11.40.254:43329)
0929 00:50:33.287095 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:33.287099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:33.287246 (+ 147us) server_negotiation.cc:407] Connection header received
0929 00:50:33.287407 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:33.287410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:33.287496 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:33.287581 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:33.288295 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.289106 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.289999 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.290308 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.290407 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.293675 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:39739: BlockingRecv error: recv got EOF from 127.11.40.194:39739 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:33 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:33.514223 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:33.502333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:40313)
0929 00:50:33.502715 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:33.502727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:33.502868 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:33.503157 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:33.503159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:33.503373 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:33.503499 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.503506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.504309 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.504314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.504960 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:33.504970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.505454 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.505457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:33.505541 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.511975 (+ 6434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:33.511989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:33.514100 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":87,"threads_started":1}
W20260929 00:50:33.514451 20624 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:33.502485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:40313 (local address 127.11.40.254:43329)
0929 00:50:33.502652 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:33.502657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:33.502805 (+ 148us) server_negotiation.cc:407] Connection header received
0929 00:50:33.502977 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:33.502980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:33.503031 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:33.503132 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:33.503625 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.504187 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.505084 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:33.505330 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:33.505445 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:33.514321 (+ 8876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:40313: BlockingRecv error: recv got EOF from 127.11.40.195:40313 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:34.107962 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:34.101852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:60113)
0929 00:50:34.102138 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:34.102152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:34.102245 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:34.102544 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:34.102546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:34.102723 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:34.102821 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.102826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.103668 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.103670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.104239 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.104246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.104705 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.104708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.104771 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.105446 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:34.105458 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:34.107835 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":112,"threads_started":1}
W20260929 00:50:34.108155 20628 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:34.102061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:60113 (local address 127.11.40.254:43329)
0929 00:50:34.102327 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:34.102331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:34.102344 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:34.102385 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:34.102387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:34.102428 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:34.102506 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:34.102997 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.103568 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.104366 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.104598 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.104692 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.107977 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:60113: BlockingRecv error: recv got EOF from 127.11.40.193:60113 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:34.300212 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:34.294594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:50843)
0929 00:50:34.294859 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:34.294873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:34.294978 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:34.295201 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:34.295203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:34.295393 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:34.295489 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.295495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.296405 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.296408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.297075 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.297085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.297502 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.297504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.297567 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.298131 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:34.298142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:34.300035 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260929 00:50:34.300285 20628 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:34.294651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:50843 (local address 127.11.40.254:43329)
0929 00:50:34.294807 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:34.294813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:34.294946 (+ 133us) server_negotiation.cc:407] Connection header received
0929 00:50:34.295053 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:34.295055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:34.295099 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:34.295168 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:34.295594 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.296268 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.297174 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.297401 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.297483 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.300173 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:50843: BlockingRecv error: recv got EOF from 127.11.40.194:50843 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:34 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:34.521492 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:34.515137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:52417)
0929 00:50:34.515556 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:34.515574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:34.515743 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:34.516043 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:34.516046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:34.516259 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:34.516355 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.516361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.517211 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.517216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.517785 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:34.517793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.518399 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.518403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:34.518527 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.519170 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:34.519186 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:34.521374 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
W20260929 00:50:34.521620 20628 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:34.515291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52417 (local address 127.11.40.254:43329)
0929 00:50:34.515531 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:34.515536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:34.515672 (+ 136us) server_negotiation.cc:407] Connection header received
0929 00:50:34.515871 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:34.515879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:34.515926 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:34.516020 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:34.516478 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.517099 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.517918 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:34.518171 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:34.518278 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:34.521490 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:52417: BlockingRecv error: recv got EOF from 127.11.40.195:52417 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:35.115381 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:35.108877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:46969)
0929 00:50:35.109194 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:35.109208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:35.109348 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:35.109605 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:35.109607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:35.109794 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:35.109894 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.109899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.110774 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.110778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.111623 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.111633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.112233 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.112236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.112317 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.113041 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:35.113059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:35.115203 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":131,"threads_started":1}
W20260929 00:50:35.115525 20632 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:35.109065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:46969 (local address 127.11.40.254:43329)
0929 00:50:35.109329 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:35.109336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:35.109352 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:35.109440 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:35.109442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:35.109488 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:35.109569 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:35.110025 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.110645 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.111773 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.112103 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.112275 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.115393 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:46969: BlockingRecv error: recv got EOF from 127.11.40.193:46969 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":64,"threads_started":1}
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:35.307487 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:35.301431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:47571)
0929 00:50:35.301783 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:35.301805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:35.301955 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:35.302248 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:35.302251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:35.302480 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:35.302571 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.302576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.303349 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.303352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.303939 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.303948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.304501 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.304504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.304607 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.305257 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:35.305275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:35.307349 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
W20260929 00:50:35.307659 20632 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:35.301581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:47571 (local address 127.11.40.254:43329)
0929 00:50:35.301749 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:35.301753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:35.301903 (+ 150us) server_negotiation.cc:407] Connection header received
0929 00:50:35.302072 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:35.302075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:35.302130 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:35.302225 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:35.302696 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.303233 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.304066 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.304362 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.304467 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.307499 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:47571: BlockingRecv error: recv got EOF from 127.11.40.194:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:35 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:35.528793 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:35.522595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:60593)
0929 00:50:35.522905 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:35.522919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:35.523042 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:35.523397 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:35.523403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:35.523618 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:35.523718 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.523724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.524658 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.524663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.525399 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:35.525410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.525944 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.525947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:35.526028 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.526629 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:35.526640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:35.528646 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":79,"threads_started":1}
W20260929 00:50:35.529030 20632 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:35.522701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:60593 (local address 127.11.40.254:43329)
0929 00:50:35.522899 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:35.522903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:35.523004 (+ 101us) server_negotiation.cc:407] Connection header received
0929 00:50:35.523135 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:35.523139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:35.523202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:35.523297 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:35.523860 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.524529 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.525555 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:35.525805 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:35.525911 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:35.528851 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:60593: BlockingRecv error: recv got EOF from 127.11.40.195:60593 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:36.122367 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:36.116394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:37353)
0929 00:50:36.116693 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:36.116706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:36.116793 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:36.117110 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:36.117112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:36.117343 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:36.117498 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.117504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.118485 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.118490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.119053 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.119061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.119550 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.119554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.119647 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.120115 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:36.120125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:36.122170 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260929 00:50:36.122428 20636 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:36.116476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:37353 (local address 127.11.40.254:43329)
0929 00:50:36.116744 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:36.116748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:36.116763 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:50:36.116912 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:36.116914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:36.116957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:36.117079 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:36.117621 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.118378 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.119174 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.119424 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.119535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.122304 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:37353: BlockingRecv error: recv got EOF from 127.11.40.193:37353 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":62,"threads_started":1}
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:36.314700 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:36.308535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:40713)
0929 00:50:36.308893 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:36.308912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:36.309024 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:36.309379 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:36.309381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:36.309684 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:36.309833 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.309840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.310858 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.310861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.311440 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.311448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.311888 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.311890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.311954 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.312544 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:36.312556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:36.314572 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
W20260929 00:50:36.314867 20636 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:36.308638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:40713 (local address 127.11.40.254:43329)
0929 00:50:36.308897 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:36.308902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:36.309010 (+ 108us) server_negotiation.cc:407] Connection header received
0929 00:50:36.309121 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:36.309125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:36.309182 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:36.309337 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:36.309969 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.310744 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.311560 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.311780 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.311870 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.314726 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:40713: BlockingRecv error: recv got EOF from 127.11.40.194:40713 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:36 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:36.536325 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:36.529805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:59365)
0929 00:50:36.530209 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:36.530223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:36.530322 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:36.530589 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:36.530591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:36.530791 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:36.530900 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.530907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.531798 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.531800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.532370 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:36.532380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.533047 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.533058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:36.533139 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.533793 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:36.533809 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:36.536097 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":219,"threads_started":1}
W20260929 00:50:36.536440 20636 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:36.530073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:59365 (local address 127.11.40.254:43329)
0929 00:50:36.530222 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:36.530225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:36.530337 (+ 112us) server_negotiation.cc:407] Connection header received
0929 00:50:36.530386 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:36.530388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:36.530433 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:36.530511 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:36.531065 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.531636 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.532514 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:36.532755 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:36.532890 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:36.536302 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:59365: BlockingRecv error: recv got EOF from 127.11.40.195:59365 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:37.129523 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:37.123299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:34405)
0929 00:50:37.123623 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:37.123637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:37.123734 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:37.124076 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:37.124079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:37.124264 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:37.124365 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.124370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.125369 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.125372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.125960 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.125970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.126443 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.126445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.126511 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.127059 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:37.127071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:37.129287 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260929 00:50:37.129585 20640 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:37.123423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:34405 (local address 127.11.40.254:43329)
0929 00:50:37.123677 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:37.123680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:37.123694 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:37.123854 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:37.123856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:37.123906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:37.124006 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:37.124543 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.125249 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.126081 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.126306 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.126408 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.129448 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:34405: BlockingRecv error: recv got EOF from 127.11.40.193:34405 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":64,"threads_started":1}
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:37.321395 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:37.315657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:33317)
0929 00:50:37.315939 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:37.315953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:37.316051 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:37.316465 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:37.316467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:37.316647 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:37.316746 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.316751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.317583 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.317586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.318144 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.318151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.318592 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.318595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.318656 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.319155 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:37.319166 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:37.321175 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
W20260929 00:50:37.321417 20640 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:37.315710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:33317 (local address 127.11.40.254:43329)
0929 00:50:37.315955 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:37.315958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:37.316037 (+ 79us) server_negotiation.cc:407] Connection header received
0929 00:50:37.316130 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:37.316132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:37.316175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:37.316250 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:37.316944 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.317478 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.318262 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.318492 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.318572 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.321313 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:33317: BlockingRecv error: recv got EOF from 127.11.40.194:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:37 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:37.542958 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:37.537326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:34813)
0929 00:50:37.537626 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:37.537639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:37.537782 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:37.538089 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:37.538092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:37.538344 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:37.538445 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.538451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.539263 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.539268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.539818 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:37.539826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.540345 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.540349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:37.540435 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.540985 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:37.540996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:37.542767 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
W20260929 00:50:37.542994 20640 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:37.537426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:34813 (local address 127.11.40.254:43329)
0929 00:50:37.537592 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:37.537598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:37.537727 (+ 129us) server_negotiation.cc:407] Connection header received
0929 00:50:37.537892 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:37.537895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:37.537979 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:37.538062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:37.538569 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.539145 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.539942 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:37.540205 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:37.540346 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:37.542890 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:34813: BlockingRecv error: recv got EOF from 127.11.40.195:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:38.136134 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:38.130494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:59819)
0929 00:50:38.130793 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:38.130807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:38.130904 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:38.131163 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:38.131165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:38.131347 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:38.131453 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.131458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.132248 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.132250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.133007 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.133017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.133572 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.133576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.133664 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.134193 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:38.134204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:38.135966 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":69,"threads_started":1}
W20260929 00:50:38.136211 20644 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:38.130570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59819 (local address 127.11.40.254:43329)
0929 00:50:38.130857 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:38.130861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:38.130875 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:50:38.130999 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:38.131001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:38.131045 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:38.131121 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:38.131560 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.132144 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.133146 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.133440 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.133567 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.136089 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:59819: BlockingRecv error: recv got EOF from 127.11.40.193:59819 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:38.329070 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:38.322326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:35775)
0929 00:50:38.322713 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:38.322730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:38.322897 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:38.323200 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:38.323202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:38.323429 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:38.323525 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.323531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.324309 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.324314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.324929 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.324939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.325416 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.325419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.325499 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.326481 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:38.326494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:38.328877 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":94,"threads_started":1}
W20260929 00:50:38.329313 20644 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:38.322482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:35775 (local address 127.11.40.254:43329)
0929 00:50:38.322687 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:38.322691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:38.322820 (+ 129us) server_negotiation.cc:407] Connection header received
0929 00:50:38.323018 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:38.323021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:38.323078 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:38.323176 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:38.323655 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.324188 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.325055 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.325285 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.325409 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.329187 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:35775: BlockingRecv error: recv got EOF from 127.11.40.194:35775 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:38 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:38.554391 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:38.543801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:50533)
0929 00:50:38.544098 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:38.544110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:38.544210 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:38.544481 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:38.544483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:38.544661 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:38.544755 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.544760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.545567 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.545569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.546134 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:38.546141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.546587 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.546590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:38.546652 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.552197 (+ 5545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:38.552210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:38.554266 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
W20260929 00:50:38.554716 20644 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:38.543970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:50533 (local address 127.11.40.254:43329)
0929 00:50:38.544106 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:38.544110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:38.544184 (+ 74us) server_negotiation.cc:407] Connection header received
0929 00:50:38.544303 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:38.544306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:38.544351 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:38.544440 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:38.544928 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.545465 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.546257 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:38.546478 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:38.546561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:38.554571 (+ 8010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:50533: BlockingRecv error: recv got EOF from 127.11.40.195:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:39.142714 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:39.137046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:45779)
0929 00:50:39.137334 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:39.137347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:39.137442 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:39.137690 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:39.137692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:39.137876 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:39.137990 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.137995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.138805 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.138808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.139393 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.139402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.139850 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.139853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.139925 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.140462 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:39.140474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:39.142591 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260929 00:50:39.142896 20648 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:39.137101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:45779 (local address 127.11.40.254:43329)
0929 00:50:39.137374 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:39.137378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:39.137390 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:50:39.137533 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:39.137535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:39.137577 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:39.137653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:39.138101 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.138657 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.139506 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.139727 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.139804 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.142752 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:45779: BlockingRecv error: recv got EOF from 127.11.40.193:45779 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":58,"threads_started":1}
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:39.335394 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:39.329873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:48229)
0929 00:50:39.330149 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:39.330163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:39.330250 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:39.330494 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:39.330497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:39.330811 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:39.330915 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.330921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.331725 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.331727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.332312 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.332319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.332765 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.332768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.332857 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.333438 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:39.333450 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:39.335257 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
W20260929 00:50:39.335564 20648 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:39.330066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:48229 (local address 127.11.40.254:43329)
0929 00:50:39.330208 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:39.330211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:39.330224 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:39.330336 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:39.330338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:39.330387 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:39.330460 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:39.331061 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.331617 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.332433 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.332661 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.332747 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.335434 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:48229: BlockingRecv error: recv got EOF from 127.11.40.194:48229 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:39 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:39.562417 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:39.555458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:54731)
0929 00:50:39.555901 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:39.555920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:39.556064 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:39.556435 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:39.556439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:39.556705 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:39.556882 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.556892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.557780 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.557783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.558367 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:39.558376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.559012 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.559016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:39.559100 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.559763 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:39.559778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:39.562262 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":138,"threads_started":1}
W20260929 00:50:39.562690 20648 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:39.555642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:54731 (local address 127.11.40.254:43329)
0929 00:50:39.555840 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:39.555847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:39.556085 (+ 238us) server_negotiation.cc:407] Connection header received
0929 00:50:39.556176 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:39.556179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:39.556248 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:39.556345 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:39.557019 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.557591 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.558540 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:39.558851 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:39.558962 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:39.562551 (+ 3589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:54731: BlockingRecv error: recv got EOF from 127.11.40.195:54731 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:40.152233 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:40.143999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:60131)
0929 00:50:40.144376 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:40.144390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:40.144493 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:40.144876 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:40.144879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:40.145242 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:40.145376 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.145384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.146893 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.146896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.147468 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.147477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.148307 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.148310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.148390 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.149216 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:40.149228 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:40.152055 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
W20260929 00:50:40.152535 20652 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:40.144019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:60131 (local address 127.11.40.254:43329)
0929 00:50:40.144518 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:40.144522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:40.144538 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:50:40.144582 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:40.144584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:40.144626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:40.144725 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:40.145923 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.146738 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.147832 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.148087 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.148235 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.152375 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:60131: BlockingRecv error: recv got EOF from 127.11.40.193:60131 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"spinlock_wait_cycles":11008,"thread_start_us":272,"threads_started":1}
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:40.342471 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:40.336273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:38121)
0929 00:50:40.336699 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:40.336715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:40.336873 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:40.337124 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:40.337126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:40.337307 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:40.337448 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.337454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.338195 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.338198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.338778 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.338786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.339202 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.339204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.339266 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.339945 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:40.339959 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:40.342292 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":85,"threads_started":1}
W20260929 00:50:40.342628 20652 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:40.336396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:38121 (local address 127.11.40.254:43329)
0929 00:50:40.336546 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:40.336550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:40.336823 (+ 273us) server_negotiation.cc:407] Connection header received
0929 00:50:40.336977 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:40.336979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:40.337024 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:40.337089 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:40.337549 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.338096 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.338884 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.339103 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.339184 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.342497 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:38121: BlockingRecv error: recv got EOF from 127.11.40.194:38121 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:40 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.195@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:40.569234 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:40.563230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.195:41851)
0929 00:50:40.563552 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:40.563575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:40.563710 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:40.564012 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:40.564015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:40.564245 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:40.564341 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.564348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.565209 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.565215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.565839 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:40.565849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.566384 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.566388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:40.566501 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.567085 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:40.567097 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:40.569047 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
W20260929 00:50:40.569294 20652 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:40.563309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:41851 (local address 127.11.40.254:43329)
0929 00:50:40.563506 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:40.563511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:40.563665 (+ 154us) server_negotiation.cc:407] Connection header received
0929 00:50:40.563826 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:40.563829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:40.563890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:40.563986 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:40.564469 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.565094 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.565969 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:40.566244 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:40.566378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:40.569177 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.195:41851: BlockingRecv error: recv got EOF from 127.11.40.195:41851 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 00:50:41 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:41 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.193@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:41.159287 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:41.153283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.193:52849)
0929 00:50:41.153587 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:41.153600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:41.153686 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:41.153940 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:41.153942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:41.154121 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:41.154216 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.154222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.154999 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.155002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.155581 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.155589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.156077 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.156081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.156184 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.156793 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:41.156859 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:41.159161 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
W20260929 00:50:41.159443 20656 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:41.153405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:52849 (local address 127.11.40.254:43329)
0929 00:50:41.153644 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:41.153648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:41.153661 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:50:41.153778 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:41.153780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:41.153823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:41.153923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:41.154316 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.154842 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.155687 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.155939 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.156021 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.159296 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.193:52849: BlockingRecv error: recv got EOF from 127.11.40.193:52849 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
Sep 29 00:50:41 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:50:41 dist-test-slave-5309 krb5kdc[19598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.40.194@KRBTEST.COM for oryx/127.11.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:50:41.350577 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:41.343762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43329 (local address 127.11.40.194:55285)
0929 00:50:41.344052 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:41.344068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:41.344182 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:41.344530 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:41.344533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:41.344798 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:41.344950 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.344972 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.346177 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.346182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.346985 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.346995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.347489 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.347493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.347579 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.348073 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:41.348084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:41.350363 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.40.254:43329: Server oryx/127.11.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260929 00:50:41.350668 20656 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:41.343876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:55285 (local address 127.11.40.254:43329)
0929 00:50:41.344033 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:41.344038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:41.344144 (+ 106us) server_negotiation.cc:407] Connection header received
0929 00:50:41.344359 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:41.344363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:41.344418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:41.344499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:41.345110 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.346049 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.347113 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.347363 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.347473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.350520 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.40.194:55285: BlockingRecv error: recv got EOF from 127.11.40.194:55285 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:50:41.431449 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20181
I20260929 00:50:41.437361 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20333
I20260929 00:50:41.443346 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20465
I20260929 00:50:41.449995 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20111
2026-09-29T00:50:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18072 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T00:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:41Z Disabled control of system clock
I20260929 00:50:41.481675 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34245
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:50:41.594071 20661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:41.594383 20661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:41.594421 20661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:41.598490 20661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:41.598562 20661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:41.598582 20661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:41.598600 20661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:41.598615 20661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:41.604635 20661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:41.605960 20661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:41.607487 20661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:41.617986 20667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:41.617973 20666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:41.618023 20661 server_base.cc:1034] running on GCE node
W20260929 00:50:41.617976 20669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:41.618912 20661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:41.620234 20661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:41.621409 20661 hybrid_clock.cc:648] HybridClock initialized: now 1790643041621372 us; error 52 us; skew 500 ppm
I20260929 00:50:41.623627 20661 webserver.cc:460] Webserver started at http://127.11.40.254:44461/ using document root <none> and password file <none>
I20260929 00:50:41.624295 20661 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:41.624367 20661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:41.624620 20661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:41.627053 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "06d877c5bb3c43f7b974e3d0b43aa85f"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "93911650dcfab041bcb44b2a431d2d21"
server_key_iv: "757fc91027031f17c7d94e84b8d6fbeb"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.627689 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "06d877c5bb3c43f7b974e3d0b43aa85f"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "93911650dcfab041bcb44b2a431d2d21"
server_key_iv: "757fc91027031f17c7d94e84b8d6fbeb"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.631915 20661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:41.634681 20675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:41.636094 20661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:50:41.636245 20661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "06d877c5bb3c43f7b974e3d0b43aa85f"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "93911650dcfab041bcb44b2a431d2d21"
server_key_iv: "757fc91027031f17c7d94e84b8d6fbeb"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.636396 20661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:41.681427 20661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:41.682274 20661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:41.682512 20661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:41.690702 20661 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34245
I20260929 00:50:41.690722 20727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34245 every 8 connection(s)
I20260929 00:50:41.691815 20661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:41.695212 20728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:41.698525 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20661
I20260929 00:50:41.698661 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:41.698935 11427 external_mini_cluster.cc:1428] Setting key b9bb3c7af6d09a6b969e61006937070b
I20260929 00:50:41.701424 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f: Bootstrap starting.
I20260929 00:50:41.703691 20728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:41.704460 20728 log.cc:824] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:41.706512 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f: No bootstrap required, opened a new log
I20260929 00:50:41.708442 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:41.700421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.11.40.254:34245)
0929 00:50:41.701181 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:41.701202 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:41.701239 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:50:41.701914 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:41.701933 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:41.702334 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:41.702671 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:41.703118 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.704143 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.705146 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.705627 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.705805 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.706696 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:41.706701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:41.706714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:41.707104 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:41.707158 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:41.707177 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:41.707308 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:41.707470 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 00:50:41.707724 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260929 00:50:41.710552 20728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } }
I20260929 00:50:41.710860 20728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:41.710927 20728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d877c5bb3c43f7b974e3d0b43aa85f, State: Initialized, Role: FOLLOWER
I20260929 00:50:41.711323 20728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } }
I20260929 00:50:41.711447 20728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:41.711498 20728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:41.711568 20728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:41.712453 20728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } }
I20260929 00:50:41.712749 20728 leader_election.cc:302] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d877c5bb3c43f7b974e3d0b43aa85f; no voters:
I20260929 00:50:41.713122 20728 leader_election.cc:288] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:41.713249 20733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:41.713519 20733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [term 1 LEADER]: Becoming Leader. State: Replica: 06d877c5bb3c43f7b974e3d0b43aa85f, State: Running, Role: LEADER
I20260929 00:50:41.713932 20733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } }
I20260929 00:50:41.714515 20728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:41.715644 20734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } } }
I20260929 00:50:41.715775 20734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:41.716179 20742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:41.716209 20735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d877c5bb3c43f7b974e3d0b43aa85f. Latest consensus state: current_term: 1 leader_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d877c5bb3c43f7b974e3d0b43aa85f" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34245 } } }
I20260929 00:50:41.716382 20735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:41.719266 20742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:41.725557 20742 catalog_manager.cc:1347] Generated new cluster ID: d90a7ae74641475690db9b6f391da78b
I20260929 00:50:41.725646 20742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:41.743554 20742 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:41.745062 20742 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:41.755184 20742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d877c5bb3c43f7b974e3d0b43aa85f: Generated new TSK 0
I20260929 00:50:41.756079 20742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:41.765669 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--builtin_ntp_servers=127.11.40.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 00:50:41.894619 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:41.894973 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:41.895056 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:41.899747 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:41.899856 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:41.900012 20752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:41.905506 20752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:41.906745 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:41.908203 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:41.918705 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:41.918725 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:41.918744 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:41.919011 20752 server_base.cc:1034] running on GCE node
I20260929 00:50:41.919535 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:41.920255 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:41.921486 20752 hybrid_clock.cc:648] HybridClock initialized: now 1790643041921457 us; error 45 us; skew 500 ppm
I20260929 00:50:41.923736 20752 webserver.cc:460] Webserver started at http://127.11.40.193:37093/ using document root <none> and password file <none>
I20260929 00:50:41.924463 20752 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:41.924579 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:41.924933 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:41.927282 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "c50670019a3e4c419203e7354d1981d1"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "2cf763e3f337ecede92990fef436b495"
server_key_iv: "f0cc62234797c38e1b0d374caa7b708e"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.927945 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "c50670019a3e4c419203e7354d1981d1"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "2cf763e3f337ecede92990fef436b495"
server_key_iv: "f0cc62234797c38e1b0d374caa7b708e"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.932354 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:50:41.935577 20766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:41.937178 20752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:41.937338 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "c50670019a3e4c419203e7354d1981d1"
format_stamp: "Formatted at 2026-09-29 00:50:41 on dist-test-slave-5309"
server_key: "2cf763e3f337ecede92990fef436b495"
server_key_iv: "f0cc62234797c38e1b0d374caa7b708e"
server_key_version: "encryptionkey@0"
I20260929 00:50:41.937475 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:41.957346 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:41.958216 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:41.958470 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:41.959178 20752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:41.960294 20752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:41.960374 20752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:41.960455 20752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:41.960513 20752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:41.970547 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43869 every 8 connection(s)
I20260929 00:50:41.970525 20752 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43869
I20260929 00:50:41.971638 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:41.973860 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20752
I20260929 00:50:41.973965 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:41.974272 11427 external_mini_cluster.cc:1428] Setting key 06dd49c9d91dc6c7c303bad4de1c9ebf
I20260929 00:50:41.976481 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--builtin_ntp_servers=127.11.40.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:50:41.983155 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:41.973774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:40879 (local address 127.11.40.254:34245)
0929 00:50:41.973952 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:41.973958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:41.975299 (+ 1341us) server_negotiation.cc:407] Connection header received
0929 00:50:41.977071 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:41.977078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:41.977142 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:41.977246 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:41.978381 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.978962 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.980780 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.981083 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.981169 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.982603 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:41.982607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:41.982610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:41.982625 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:41.982647 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:41.982657 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:41.982747 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:41.982945 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 00:50:41.983008 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:50:41.984620 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:41.974337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34245 (local address 127.11.40.193:40879)
0929 00:50:41.975151 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:41.975212 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:41.976789 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:41.977446 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:41.977453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:41.977541 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:41.978217 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.978237 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.979100 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.979107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.980638 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:41.980653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:41.981256 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:41.981263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:41.981526 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:41.982214 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:41.982218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:41.982284 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:41.982395 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:41.982770 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:41.982782 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:41.982789 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:50:41.982940 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 00:50:41.983418 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260929 00:50:41.986276 20880 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34245
I20260929 00:50:41.986557 20880 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:41.987118 20880 heartbeater.cc:511] Master 127.11.40.254:34245 requested a full tablet report, sending...
I20260929 00:50:41.988727 20692 ts_manager.cc:194] Registered new tserver with Master: c50670019a3e4c419203e7354d1981d1 (127.11.40.193:43869)
I20260929 00:50:41.989982 20692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.193:40879
W20260929 00:50:42.093441 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:42.093799 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:42.093880 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:42.098119 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:42.098229 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:42.098335 20883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:42.103382 20883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:42.104621 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:42.106158 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:42.117205 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.117406 20883 server_base.cc:1034] running on GCE node
W20260929 00:50:42.117230 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.117185 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.118026 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:42.118798 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:42.120013 20883 hybrid_clock.cc:648] HybridClock initialized: now 1790643042119967 us; error 55 us; skew 500 ppm
I20260929 00:50:42.122511 20883 webserver.cc:460] Webserver started at http://127.11.40.194:37449/ using document root <none> and password file <none>
I20260929 00:50:42.123162 20883 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:42.123260 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:42.123512 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:42.126129 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "b23e965054fd4f2daa6f05c99f5ee151"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "e3637333ab4b366ffc5bd5fce83c5a1b"
server_key_iv: "feb6dd65bfc901d6869735b800909f62"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.126814 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "b23e965054fd4f2daa6f05c99f5ee151"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "e3637333ab4b366ffc5bd5fce83c5a1b"
server_key_iv: "feb6dd65bfc901d6869735b800909f62"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.131279 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:42.134064 20897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.135402 20883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:50:42.135521 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "b23e965054fd4f2daa6f05c99f5ee151"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "e3637333ab4b366ffc5bd5fce83c5a1b"
server_key_iv: "feb6dd65bfc901d6869735b800909f62"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.135635 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:42.182757 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:42.183583 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:42.183815 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:42.184511 20883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:42.185700 20883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:42.185777 20883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.185853 20883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:42.185899 20883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.195106 20883 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:36791
I20260929 00:50:42.195128 21010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:36791 every 8 connection(s)
I20260929 00:50:42.196096 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:42.205109 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 20883
I20260929 00:50:42.205256 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:42.205634 11427 external_mini_cluster.cc:1428] Setting key c949591981611c45d671ffd6c2167031
I20260929 00:50:42.208017 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--builtin_ntp_servers=127.11.40.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:50:42.210881 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.198129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:38427 (local address 127.11.40.254:34245)
0929 00:50:42.198308 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:42.198311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:42.199514 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 00:50:42.201011 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:42.201014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:42.201078 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:42.201206 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:42.202185 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.202735 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.203921 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.204154 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.204271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:42.210351 (+ 6080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:42.210355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:42.210357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:42.210371 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:42.210395 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:42.210404 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:42.210467 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:42.210687 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 00:50:42.210757 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:50:42.212276 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.198564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34245 (local address 127.11.40.194:38427)
0929 00:50:42.199331 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:42.199377 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:42.200750 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:42.201392 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:42.201399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:42.201491 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:42.202032 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:42.202045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.202876 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.202884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:42.203792 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:42.203801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.204319 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.204323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:42.204490 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:42.209911 (+ 5421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:42.209915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:42.209997 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:42.210114 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:42.210509 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:42.210521 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:42.210529 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:50:42.210689 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 00:50:42.211072 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":134,"threads_started":1}
I20260929 00:50:42.213686 21011 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34245
I20260929 00:50:42.214035 21011 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:42.214736 21011 heartbeater.cc:511] Master 127.11.40.254:34245 requested a full tablet report, sending...
I20260929 00:50:42.216229 20692 ts_manager.cc:194] Registered new tserver with Master: b23e965054fd4f2daa6f05c99f5ee151 (127.11.40.194:36791)
I20260929 00:50:42.217024 20692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.194:38427
W20260929 00:50:42.327845 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:42.328123 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:42.328161 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:42.332226 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:42.332289 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:42.332345 21014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:42.337430 21014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:42.338687 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:42.340265 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:42.352007 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.351994 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.351998 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.352043 21014 server_base.cc:1034] running on GCE node
I20260929 00:50:42.352735 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:42.353489 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:42.354676 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790643042354621 us; error 57 us; skew 500 ppm
I20260929 00:50:42.356634 21014 webserver.cc:460] Webserver started at http://127.11.40.195:40173/ using document root <none> and password file <none>
I20260929 00:50:42.357291 21014 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:42.357354 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:42.357573 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:42.359725 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "2e7bb08c6422451aa5bbc765696e9a64"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "51ef6513b9aff99ed1cc3f04e514dfe9"
server_key_iv: "521adbadc5bf595d3e5eb81f05f59dc5"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.360297 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "2e7bb08c6422451aa5bbc765696e9a64"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "51ef6513b9aff99ed1cc3f04e514dfe9"
server_key_iv: "521adbadc5bf595d3e5eb81f05f59dc5"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.364126 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:42.366897 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.368211 21014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:50:42.368353 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "2e7bb08c6422451aa5bbc765696e9a64"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "51ef6513b9aff99ed1cc3f04e514dfe9"
server_key_iv: "521adbadc5bf595d3e5eb81f05f59dc5"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.368479 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:42.417568 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:42.418313 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:42.418533 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:42.419318 21014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:42.420621 21014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:42.420678 21014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.420742 21014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:42.420773 21014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.430019 21014 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:42175
I20260929 00:50:42.430037 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:42175 every 8 connection(s)
I20260929 00:50:42.430955 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:42.436338 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21014
I20260929 00:50:42.436494 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:42.436879 11427 external_mini_cluster.cc:1428] Setting key 7bc54f399385d3b4fbe6152ecf3ef5c3
I20260929 00:50:42.441303 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.432778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:46711 (local address 127.11.40.254:34245)
0929 00:50:42.432943 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:42.432946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:42.434007 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 00:50:42.435400 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:42.435404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:42.435465 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:42.435572 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:42.436709 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.437417 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.439074 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.439380 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.439465 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:42.440701 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:42.440704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:42.440706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:42.440718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:42.440736 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:42.440744 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:42.440859 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:42.441057 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 00:50:42.441128 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:50:42.442478 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.433189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34245 (local address 127.11.40.195:46711)
0929 00:50:42.433853 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:42.433895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:42.435137 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:42.435742 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:42.435749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:42.435841 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:42.436337 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:42.436351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.437542 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.437545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:42.438866 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:42.438884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:42.439540 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:42.439543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:42.439720 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:42.440445 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:42.440446 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:42.440489 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:42.440559 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:42.440889 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:42.440898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:42.440905 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:50:42.441065 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 00:50:42.441418 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260929 00:50:42.443670 21142 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34245
I20260929 00:50:42.443955 21142 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:42.444522 21142 heartbeater.cc:511] Master 127.11.40.254:34245 requested a full tablet report, sending...
I20260929 00:50:42.445676 20692 ts_manager.cc:194] Registered new tserver with Master: 2e7bb08c6422451aa5bbc765696e9a64 (127.11.40.195:42175)
I20260929 00:50:42.446333 20692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.195:46711
I20260929 00:50:42.451166 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:50:42.454195 20731 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:42.453324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.11.40.254:34245)
0929 00:50:42.453466 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:42.453469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:42.453632 (+ 163us) server_negotiation.cc:407] Connection header received
0929 00:50:42.453730 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:42.453732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:42.453771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:42.453837 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:42.454041 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42046: BlockingRecv error: recv got EOF from 127.0.0.1:42046 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:50:42.454681 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20752
I20260929 00:50:42.461086 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20883
I20260929 00:50:42.466509 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21014
I20260929 00:50:42.471808 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 20661
2026-09-29T00:50:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1019 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T00:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:42Z Disabled control of system clock
I20260929 00:50:42.501412 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39171
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:50:42.613921 21153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:42.614199 21153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:42.614238 21153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:42.618340 21153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:42.618412 21153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:42.618427 21153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:42.618446 21153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:42.618462 21153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:42.623279 21153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:39171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:42.624423 21153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:42.625866 21153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:42.634397 21159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.634418 21161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.634560 21153 server_base.cc:1034] running on GCE node
W20260929 00:50:42.634757 21158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.635331 21153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:42.636446 21153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:42.637632 21153 hybrid_clock.cc:648] HybridClock initialized: now 1790643042637590 us; error 56 us; skew 500 ppm
I20260929 00:50:42.639601 21153 webserver.cc:460] Webserver started at http://127.11.40.254:43355/ using document root <none> and password file <none>
I20260929 00:50:42.640204 21153 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:42.640300 21153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:42.640525 21153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:42.642755 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "3792ba4e277142eaad762de3c2e30658"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "ec8b5eff1cd62de8e1867871e2e7e0cb"
server_key_iv: "86d133b5d1b53ab47800da376ee3a4f0"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.643402 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "3792ba4e277142eaad762de3c2e30658"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "ec8b5eff1cd62de8e1867871e2e7e0cb"
server_key_iv: "86d133b5d1b53ab47800da376ee3a4f0"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.647569 21153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:50:42.651407 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.653266 21153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:42.653411 21153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "3792ba4e277142eaad762de3c2e30658"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "ec8b5eff1cd62de8e1867871e2e7e0cb"
server_key_iv: "86d133b5d1b53ab47800da376ee3a4f0"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.653563 21153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:42.659104 21153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:42.659739 21153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:42.659929 21153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:42.666916 21153 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:39171
I20260929 00:50:42.666939 21219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:39171 every 8 connection(s)
I20260929 00:50:42.668376 21153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:42.671717 21220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:42.677381 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21153
I20260929 00:50:42.677419 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658: Bootstrap starting.
I20260929 00:50:42.677515 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:42.677776 11427 external_mini_cluster.cc:1428] Setting key c6a174d536fc07c2cbac525bc8cdcae1
I20260929 00:50:42.679996 21220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:42.680744 21220 log.cc:824] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:42.682878 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658: No bootstrap required, opened a new log
I20260929 00:50:42.683432 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.679186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.11.40.254:39171)
0929 00:50:42.679811 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:42.679827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:42.679861 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:50:42.680544 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:42.680563 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:42.681032 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:42.681357 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:42.682099 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:42.682104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:42.682115 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:42.682125 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:42.682176 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:42.682191 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:42.682308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:42.682436 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 00:50:42.682669 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":139,"threads_started":1}
I20260929 00:50:42.686585 21220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } }
I20260929 00:50:42.686892 21220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:42.686968 21220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3792ba4e277142eaad762de3c2e30658, State: Initialized, Role: FOLLOWER
I20260929 00:50:42.687453 21220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } }
I20260929 00:50:42.687584 21220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:42.687625 21220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:42.687695 21220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:42.688571 21220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } }
I20260929 00:50:42.688959 21220 leader_election.cc:302] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3792ba4e277142eaad762de3c2e30658; no voters:
I20260929 00:50:42.689234 21220 leader_election.cc:288] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:42.689359 21225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:42.689626 21225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [term 1 LEADER]: Becoming Leader. State: Replica: 3792ba4e277142eaad762de3c2e30658, State: Running, Role: LEADER
I20260929 00:50:42.689940 21225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } }
I20260929 00:50:42.690496 21220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:42.691584 21227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3792ba4e277142eaad762de3c2e30658. Latest consensus state: current_term: 1 leader_uuid: "3792ba4e277142eaad762de3c2e30658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } } }
I20260929 00:50:42.691720 21227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:42.692009 21226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3792ba4e277142eaad762de3c2e30658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3792ba4e277142eaad762de3c2e30658" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39171 } } }
I20260929 00:50:42.692101 21226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:42.692212 21234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:42.695461 21234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:42.701233 21234 catalog_manager.cc:1347] Generated new cluster ID: 19c8327202ff4429ab9aac61159bd423
I20260929 00:50:42.701320 21234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:42.724316 21234 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:42.724712 21234 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:42.736547 21234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3792ba4e277142eaad762de3c2e30658: Generated new TSK 0
I20260929 00:50:42.737489 21234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:50:42.750437 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--builtin_ntp_servers=127.11.40.212:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:50:42.862932 21244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:42.863276 21244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:42.863353 21244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:42.867615 21244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:42.867708 21244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:42.867801 21244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:42.872992 21244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:42.874151 21244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:42.875546 21244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:42.885432 21249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.885363 21250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:42.885363 21252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:42.885725 21244 server_base.cc:1034] running on GCE node
I20260929 00:50:42.886165 21244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:42.886884 21244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:42.888063 21244 hybrid_clock.cc:648] HybridClock initialized: now 1790643042888025 us; error 46 us; skew 500 ppm
I20260929 00:50:42.890211 21244 webserver.cc:460] Webserver started at http://127.11.40.193:35509/ using document root <none> and password file <none>
I20260929 00:50:42.890877 21244 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:42.890950 21244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:42.891248 21244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:42.893652 21244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "743c240b08c74df89cedea721bedd188"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "7ffd22ecd752c166062dc9c0dc69732f"
server_key_iv: "d7a2e06b78bfd7c628866a729f0edac7"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.894258 21244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "743c240b08c74df89cedea721bedd188"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "7ffd22ecd752c166062dc9c0dc69732f"
server_key_iv: "d7a2e06b78bfd7c628866a729f0edac7"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.898211 21244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 00:50:42.900983 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.902355 21244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:50:42.902504 21244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "743c240b08c74df89cedea721bedd188"
format_stamp: "Formatted at 2026-09-29 00:50:42 on dist-test-slave-5309"
server_key: "7ffd22ecd752c166062dc9c0dc69732f"
server_key_iv: "d7a2e06b78bfd7c628866a729f0edac7"
server_key_version: "encryptionkey@0"
I20260929 00:50:42.902630 21244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:42.907454 21244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:42.908085 21244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:42.908277 21244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:42.908938 21244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:42.909875 21244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:42.909921 21244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.909987 21244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:42.910017 21244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:42.919072 21244 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45913
I20260929 00:50:42.919094 21371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45913 every 8 connection(s)
I20260929 00:50:42.920017 21244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:42.926422 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.921964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:43347 (local address 127.11.40.254:39171)
0929 00:50:42.922131 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:42.922135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:42.923208 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 00:50:42.924603 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:42.924606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:42.924652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:42.924795 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:42.925929 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:42.925932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:42.925935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:42.925947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:42.925977 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:42.925987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:42.926090 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:42.926234 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 00:50:42.926282 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:50:42.927214 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21244
I20260929 00:50:42.927315 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:42.927333 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:42.922366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39171 (local address 127.11.40.193:43347)
0929 00:50:42.923048 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:42.923092 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:42.924382 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:42.924993 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:42.925000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:42.925091 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:42.925660 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:42.925662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:42.925707 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:42.925776 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:42.926071 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:42.926080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:42.926086 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:50:42.926308 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 00:50:42.926557 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260929 00:50:42.927671 11427 external_mini_cluster.cc:1428] Setting key 55d708c6fd78eb4c2c07e3eaf6435905
I20260929 00:50:42.928746 21372 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39171
I20260929 00:50:42.929029 21372 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:42.929487 21372 heartbeater.cc:511] Master 127.11.40.254:39171 requested a full tablet report, sending...
I20260929 00:50:42.929785 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--builtin_ntp_servers=127.11.40.212:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 00:50:42.931066 21178 ts_manager.cc:194] Registered new tserver with Master: 743c240b08c74df89cedea721bedd188 (127.11.40.193:45913)
W20260929 00:50:43.041792 21375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:43.042061 21375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:43.042101 21375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:43.046101 21375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:43.046154 21375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:43.046211 21375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:43.050902 21375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:43.051964 21375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:43.053345 21375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:43.063249 21383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:43.063318 21380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:43.063233 21381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:43.063400 21375 server_base.cc:1034] running on GCE node
I20260929 00:50:43.063879 21375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:43.064546 21375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:43.065733 21375 hybrid_clock.cc:648] HybridClock initialized: now 1790643043065709 us; error 50 us; skew 500 ppm
I20260929 00:50:43.067683 21375 webserver.cc:460] Webserver started at http://127.11.40.194:44051/ using document root <none> and password file <none>
I20260929 00:50:43.068265 21375 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:43.068385 21375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:43.068627 21375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:43.070756 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "b567a8e201334b928385291431a5201e"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "984c4566b25bbfa122884de7b0df6f87"
server_key_iv: "69c4787f31a281f95cef95b6c478b049"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.071259 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "b567a8e201334b928385291431a5201e"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "984c4566b25bbfa122884de7b0df6f87"
server_key_iv: "69c4787f31a281f95cef95b6c478b049"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.074828 21375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:50:43.077411 21389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.078624 21375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:50:43.078719 21375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "b567a8e201334b928385291431a5201e"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "984c4566b25bbfa122884de7b0df6f87"
server_key_iv: "69c4787f31a281f95cef95b6c478b049"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.078811 21375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:43.083276 21375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:43.083850 21375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:43.083992 21375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:43.084455 21375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:43.085354 21375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:43.085408 21375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.085445 21375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:43.085460 21375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.094518 21375 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:36009
I20260929 00:50:43.094543 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:36009 every 8 connection(s)
I20260929 00:50:43.095415 21375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:43.096701 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21375
I20260929 00:50:43.096863 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:43.097150 11427 external_mini_cluster.cc:1428] Setting key b2666f4c9871958b08a267cd9af545ad
I20260929 00:50:43.099189 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--builtin_ntp_servers=127.11.40.212:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 00:50:43.101991 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:43.097292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:38197 (local address 127.11.40.254:39171)
0929 00:50:43.097448 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:43.097453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:43.098734 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 00:50:43.100122 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:43.100129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:43.100178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:43.100273 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:43.101554 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:43.101557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:43.101559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:43.101570 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:43.101589 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:43.101597 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:43.101673 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:43.101841 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 00:50:43.101887 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:50:43.103169 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:43.097808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39171 (local address 127.11.40.194:38197)
0929 00:50:43.098571 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:43.098623 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:43.099913 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:43.100459 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:43.100471 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:43.100606 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:43.101219 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:43.101222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:43.101291 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:43.101386 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:43.101688 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:43.101698 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:43.101704 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:50:43.101887 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 00:50:43.102131 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":149,"threads_started":1}
I20260929 00:50:43.104363 21503 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39171
I20260929 00:50:43.104629 21503 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:43.105192 21503 heartbeater.cc:511] Master 127.11.40.254:39171 requested a full tablet report, sending...
I20260929 00:50:43.106485 21178 ts_manager.cc:194] Registered new tserver with Master: b567a8e201334b928385291431a5201e (127.11.40.194:36009)
W20260929 00:50:43.222004 21506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:43.222383 21506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:43.222431 21506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:43.226758 21506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:43.226820 21506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:43.226881 21506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:43.231943 21506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:43.233131 21506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:43.234508 21506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:50:43.244735 21506 server_base.cc:1034] running on GCE node
W20260929 00:50:43.244799 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:43.244771 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:43.244745 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:43.245378 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:43.246044 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:43.247212 21506 hybrid_clock.cc:648] HybridClock initialized: now 1790643043247192 us; error 32 us; skew 500 ppm
I20260929 00:50:43.249254 21506 webserver.cc:460] Webserver started at http://127.11.40.195:44513/ using document root <none> and password file <none>
I20260929 00:50:43.249855 21506 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:43.249926 21506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:43.250149 21506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:43.252296 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "b2cd263e1f6a4fe5bbe90d282f39c88d"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "fa51d5b687e7b55f9bab651c2751c90d"
server_key_iv: "ab22b1f04c8ec520c5bc4891839cbeca"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.252954 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "b2cd263e1f6a4fe5bbe90d282f39c88d"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "fa51d5b687e7b55f9bab651c2751c90d"
server_key_iv: "ab22b1f04c8ec520c5bc4891839cbeca"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.257031 21506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:50:43.260030 21520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.261534 21506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:50:43.261682 21506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "b2cd263e1f6a4fe5bbe90d282f39c88d"
format_stamp: "Formatted at 2026-09-29 00:50:43 on dist-test-slave-5309"
server_key: "fa51d5b687e7b55f9bab651c2751c90d"
server_key_iv: "ab22b1f04c8ec520c5bc4891839cbeca"
server_key_version: "encryptionkey@0"
I20260929 00:50:43.261816 21506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:43.266618 21506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:43.267258 21506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:43.267445 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:43.267992 21506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:43.269083 21506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:43.269153 21506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.269235 21506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:43.269269 21506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:43.277890 21506 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:37031
I20260929 00:50:43.277967 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:37031 every 8 connection(s)
I20260929 00:50:43.278828 21506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:43.285418 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:43.280864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:49973 (local address 127.11.40.254:39171)
0929 00:50:43.281004 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:43.281008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:43.282172 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 00:50:43.283622 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:43.283625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:43.283663 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:43.283746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:43.284960 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:43.284962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:43.284965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:50:43.284976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:43.284997 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:50:43.285005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:43.285085 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:43.285245 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:50:43.285284 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:50:43.286273 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:43.281267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39171 (local address 127.11.40.195:49973)
0929 00:50:43.282009 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:43.282053 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:43.283416 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:43.283900 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:43.283908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:43.284007 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:43.284605 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:50:43.284606 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:43.284649 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:50:43.284724 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:43.285073 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:43.285078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:43.285082 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:50:43.285269 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 00:50:43.285462 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260929 00:50:43.286887 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21506
I20260929 00:50:43.287002 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:50:43.287338 11427 external_mini_cluster.cc:1428] Setting key d07bff9cadcd9f75b1814f360d7be327
I20260929 00:50:43.287524 21634 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39171
I20260929 00:50:43.287779 21634 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:43.288183 21634 heartbeater.cc:511] Master 127.11.40.254:39171 requested a full tablet report, sending...
I20260929 00:50:43.289361 21179 ts_manager.cc:194] Registered new tserver with Master: b2cd263e1f6a4fe5bbe90d282f39c88d (127.11.40.195:37031)
I20260929 00:50:43.301805 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:50:43.304782 21223 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:50:43.303916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.11.40.254:39171)
0929 00:50:43.304091 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:43.304094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:43.304163 (+ 69us) server_negotiation.cc:407] Connection header received
0929 00:50:43.304270 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:43.304272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:43.304305 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:43.304372 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:43.304647 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33322: BlockingRecv error: recv got EOF from 127.0.0.1:33322 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 00:50:43.304993 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21244
I20260929 00:50:43.310333 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21375
I20260929 00:50:43.317481 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21506
I20260929 00:50:43.322750 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21153
2026-09-29T00:50:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (850 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:43 dist-test-slave-5309 krb5kdc[21644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:43 dist-test-slave-5309 krb5kdc[21644](info): set up 2 sockets
Sep 29 00:50:43 dist-test-slave-5309 krb5kdc[21644](info): commencing operation
krb5kdc: starting...
W20260929 00:50:44.375463 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:44 dist-test-slave-5309 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:50:44.504155 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41293
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:44.619113 21657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:44.619396 21657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:44.619438 21657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:44.623485 21657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:50:44.623551 21657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:44.623571 21657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:44.623587 21657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:50:44.623603 21657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:50:44.628988 21657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:44.630025 21657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:44.630990 21657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:50:44.640219 21657 server_base.cc:1034] running on GCE node
W20260929 00:50:44.640163 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:44.640162 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:44.640576 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:44.641264 21657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:44.642491 21657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:44.643687 21657 hybrid_clock.cc:648] HybridClock initialized: now 1790643044643657 us; error 46 us; skew 500 ppm
Sep 29 00:50:44 dist-test-slave-5309 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643044, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:44.653265 21657 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:50:44.654225 21657 webserver.cc:460] Webserver started at http://127.11.40.254:33077/ using document root <none> and password file <none>
I20260929 00:50:44.654778 21657 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:44.654839 21657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:44.655017 21657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:44.657171 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "d3516e37f27d4f3e97d9ca7171ff3f84"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "85414bc0cac620a6f013957e965721d1"
server_key_iv: "9f000a815004368a5d22370f7581627f"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.657701 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "d3516e37f27d4f3e97d9ca7171ff3f84"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "85414bc0cac620a6f013957e965721d1"
server_key_iv: "9f000a815004368a5d22370f7581627f"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.661523 21657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:50:44.664070 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:44.665472 21657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:50:44.665608 21657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "d3516e37f27d4f3e97d9ca7171ff3f84"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "85414bc0cac620a6f013957e965721d1"
server_key_iv: "9f000a815004368a5d22370f7581627f"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.665714 21657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:44.676638 21657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:44.679741 21657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:44.679972 21657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:44.687426 21657 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:41293
I20260929 00:50:44.687728 21724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:41293 every 8 connection(s)
I20260929 00:50:44.688504 21657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:50:44.691253 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21657
I20260929 00:50:44.691393 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:50:44.691627 21725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:44.691713 11427 external_mini_cluster.cc:1428] Setting key af6b61eae0ec0a8cda39bf54bc7d0bfb
I20260929 00:50:44.698714 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: Bootstrap starting.
I20260929 00:50:44.701627 21725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: Neither blocks nor log segments found. Creating new log.
Sep 29 00:50:44 dist-test-slave-5309 krb5kdc[21644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:44.702466 21725 log.cc:824] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:44.704718 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: No bootstrap required, opened a new log
I20260929 00:50:44.707955 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:44.693410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.11.40.254:41293)
0929 00:50:44.693960 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:44.693976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:44.694022 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:50:44.695035 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:44.695056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:44.695421 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:44.695765 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:44.696663 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:44.697874 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:44.699035 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:44.699527 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:44.699662 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:44.702834 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:44.702842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:44.702862 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:44.702927 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:44.704780 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:44.705364 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:44.705369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:44.705375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:44.705493 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:44.705789 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:44.705791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:44.705792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:44.706191 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:44.706331 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:44.706641 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 00:50:44.706974 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":141,"threads_started":1}
I20260929 00:50:44.708964 21725 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } }
I20260929 00:50:44.709249 21725 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:44.709340 21725 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3516e37f27d4f3e97d9ca7171ff3f84, State: Initialized, Role: FOLLOWER
I20260929 00:50:44.709779 21725 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } }
I20260929 00:50:44.709930 21725 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:50:44.710016 21725 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:50:44.710119 21725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:44.711057 21725 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } }
I20260929 00:50:44.711407 21725 leader_election.cc:302] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3516e37f27d4f3e97d9ca7171ff3f84; no voters:
I20260929 00:50:44.711722 21725 leader_election.cc:288] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:50:44.711869 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:44.712129 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [term 1 LEADER]: Becoming Leader. State: Replica: d3516e37f27d4f3e97d9ca7171ff3f84, State: Running, Role: LEADER
I20260929 00:50:44.712488 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } }
I20260929 00:50:44.713095 21725 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:50:44.714363 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } } }
I20260929 00:50:44.714429 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3516e37f27d4f3e97d9ca7171ff3f84. Latest consensus state: current_term: 1 leader_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3516e37f27d4f3e97d9ca7171ff3f84" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41293 } } }
I20260929 00:50:44.714498 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:44.714556 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84 [sys.catalog]: This master's current role is: LEADER
I20260929 00:50:44.714931 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:50:44.718173 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:50:44.723707 21739 catalog_manager.cc:1347] Generated new cluster ID: a4e923a16f9a4963ae6e2b38b3997877
I20260929 00:50:44.723794 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:50:44.753463 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:50:44.754781 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:50:44.761536 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: Generated new TSK 0
I20260929 00:50:44.762334 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:50:44.800179 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:44.940219 21751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:44.940495 21751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:44.940539 21751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:44.944761 21751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:44.944842 21751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:44.944913 21751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:50:44.950214 21751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:44.951298 21751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:44.952317 21751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:44.962417 21757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:44.962417 21756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:44.962419 21759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:44.962730 21751 server_base.cc:1034] running on GCE node
I20260929 00:50:44.963215 21751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:44.963986 21751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:44.965265 21751 hybrid_clock.cc:648] HybridClock initialized: now 1790643044965250 us; error 31 us; skew 500 ppm
Sep 29 00:50:44 dist-test-slave-5309 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643044, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:44.968739 21751 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:50:44.969965 21751 webserver.cc:460] Webserver started at http://127.11.40.193:39631/ using document root <none> and password file <none>
I20260929 00:50:44.970701 21751 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:44.970799 21751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:44.971047 21751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:44.973397 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "8dec35f1c4b942d48bffa4aedb9be62b"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "d967af6232b0d28adfaf77969ed13346"
server_key_iv: "e71c1f77c4ab34888c417fa3f63aeecc"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.974033 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "8dec35f1c4b942d48bffa4aedb9be62b"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "d967af6232b0d28adfaf77969ed13346"
server_key_iv: "e71c1f77c4ab34888c417fa3f63aeecc"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.978752 21751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:50:44.981808 21766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:44.983662 21751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:50:44.983862 21751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "8dec35f1c4b942d48bffa4aedb9be62b"
format_stamp: "Formatted at 2026-09-29 00:50:44 on dist-test-slave-5309"
server_key: "d967af6232b0d28adfaf77969ed13346"
server_key_iv: "e71c1f77c4ab34888c417fa3f63aeecc"
server_key_version: "encryptionkey@0"
I20260929 00:50:44.984012 21751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:45.002275 21751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:45.006181 21751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:45.006410 21751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:45.007043 21751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:45.007965 21751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:45.008033 21751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.008101 21751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:45.008148 21751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.018324 21751 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:43997
I20260929 00:50:45.018383 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:43997 every 8 connection(s)
I20260929 00:50:45.019294 21751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:50:45.028404 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21751
I20260929 00:50:45.028563 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:50:45.028968 11427 external_mini_cluster.cc:1428] Setting key f34d8548189af8a0f5855dbcb4fb196c
Sep 29 00:50:45 dist-test-slave-5309 krb5kdc[21644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643044, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:45.035583 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.021506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:41213 (local address 127.11.40.254:41293)
0929 00:50:45.021702 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.021706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.022839 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 00:50:45.024344 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.024349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.024410 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.024519 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:45.025871 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.026475 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.027676 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.027923 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.028046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.031045 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:45.031048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:45.031051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:45.031080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:45.032984 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.033583 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.033587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.033589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.033703 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.034162 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.034165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.034167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.034417 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:45.034507 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.035258 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.035452 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 00:50:45.036923 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.022100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:41293 (local address 127.11.40.193:41213)
0929 00:50:45.022635 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.022684 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.024060 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.024712 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.024720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.025189 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:45.025698 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.025713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.026617 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.026622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.027539 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.027548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.028094 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.028099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.028287 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.029014 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:45.029031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:45.030859 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:45.033116 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.033122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.033136 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.033472 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.033809 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.033815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.033819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.034028 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.034542 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:45.034552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:45.034919 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.035162 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.035603 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260929 00:50:45.038622 21880 heartbeater.cc:347] Connected to a master server at 127.11.40.254:41293
I20260929 00:50:45.038965 21880 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:45.039561 21880 heartbeater.cc:511] Master 127.11.40.254:41293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:50:45.041460 21689 ts_manager.cc:194] Registered new tserver with Master: 8dec35f1c4b942d48bffa4aedb9be62b (127.11.40.193:43997)
I20260929 00:50:45.042836 21689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:41213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:50:45.062093 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:45.176985 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:45.177268 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:45.177313 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:45.181444 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:45.181515 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:45.181576 21885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:50:45.187011 21885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:45.188155 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:45.189306 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:45.199541 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:45.199736 21885 server_base.cc:1034] running on GCE node
W20260929 00:50:45.199585 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:45.199543 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:45.200266 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:45.201030 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:45.202219 21885 hybrid_clock.cc:648] HybridClock initialized: now 1790643045202182 us; error 46 us; skew 500 ppm
Sep 29 00:50:45 dist-test-slave-5309 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:45.205076 21885 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:50:45.206028 21885 webserver.cc:460] Webserver started at http://127.11.40.194:46003/ using document root <none> and password file <none>
I20260929 00:50:45.206630 21885 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:45.206728 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:45.207023 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:45.209334 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "ae23aa372b5a419fb66ba83dd6b27eab"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "6cba1a87c0ba405802fcd2b6c5a4ecc0"
server_key_iv: "7db113540e0a83f608a8bf617c961a4d"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.209856 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "ae23aa372b5a419fb66ba83dd6b27eab"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "6cba1a87c0ba405802fcd2b6c5a4ecc0"
server_key_iv: "7db113540e0a83f608a8bf617c961a4d"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.213662 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:50:45.217223 21900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.218775 21885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:50:45.218904 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "ae23aa372b5a419fb66ba83dd6b27eab"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "6cba1a87c0ba405802fcd2b6c5a4ecc0"
server_key_iv: "7db113540e0a83f608a8bf617c961a4d"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.219055 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:45.238541 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:45.243084 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:45.243306 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:45.244005 21885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:45.245062 21885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:45.245115 21885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.245158 21885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:45.245174 21885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.255419 21885 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:43493
I20260929 00:50:45.255445 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:43493 every 8 connection(s)
I20260929 00:50:45.256420 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:50:45.259150 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 21885
I20260929 00:50:45.259284 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:50:45.259593 11427 external_mini_cluster.cc:1428] Setting key 469030adea906a7228d6f89cef8ec6ea
Sep 29 00:50:45 dist-test-slave-5309 krb5kdc[21644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:50:45.274920 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.258485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:52273 (local address 127.11.40.254:41293)
0929 00:50:45.258633 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.258637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.259625 (+ 988us) server_negotiation.cc:407] Connection header received
0929 00:50:45.261432 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.261435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.261480 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.261555 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:45.263507 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.264435 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.265870 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.266192 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.266355 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.270263 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:45.270267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:45.270272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:45.270328 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:45.272050 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.272909 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.272913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.272915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.273024 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.273568 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.273571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.273572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.273824 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:45.273915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.274612 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.274768 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:50:45.275842 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.258840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:41293 (local address 127.11.40.194:52273)
0929 00:50:45.259431 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.259496 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.261170 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.261817 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.261827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.262448 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:45.263275 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.263304 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.264612 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.264616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.265687 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.265698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.266413 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.266418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.266622 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.267559 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:45.267578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:45.269939 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:45.272208 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.272219 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.272244 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.272668 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.273159 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.273161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.273163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.273443 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.273970 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:45.273976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:45.274276 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.274495 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.274812 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:50:45.277292 22014 heartbeater.cc:347] Connected to a master server at 127.11.40.254:41293
I20260929 00:50:45.277566 22014 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:45.278111 22014 heartbeater.cc:511] Master 127.11.40.254:41293 requested a full tablet report, sending...
I20260929 00:50:45.279255 21689 ts_manager.cc:194] Registered new tserver with Master: ae23aa372b5a419fb66ba83dd6b27eab (127.11.40.194:43493)
I20260929 00:50:45.279873 21689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:52273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:50:45.297993 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:50:45.418249 22019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:50:45.418607 22019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:50:45.418654 22019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:50:45.423236 22019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:50:45.423334 22019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:50:45.423499 22019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:50:45.429495 22019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:50:45.430745 22019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:50:45.431905 22019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:50:45.443269 22024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:50:45.443248 22025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:45.443245 22019 server_base.cc:1034] running on GCE node
W20260929 00:50:45.443218 22027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:50:45.444038 22019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:50:45.444742 22019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:50:45.446007 22019 hybrid_clock.cc:648] HybridClock initialized: now 1790643045445975 us; error 46 us; skew 500 ppm
Sep 29 00:50:45 dist-test-slave-5309 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:50:45.449391 22019 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:50:45.450310 22019 webserver.cc:460] Webserver started at http://127.11.40.195:37123/ using document root <none> and password file <none>
I20260929 00:50:45.450951 22019 fs_manager.cc:359] Metadata directory not provided
I20260929 00:50:45.451050 22019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:50:45.451269 22019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:50:45.453442 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "b8666c84ba6f4bbba2eb4e712baa231f"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "c2d71eb50ea1ea34db21d841b14b6ec3"
server_key_iv: "5af8251886011809be5311ecff25ba15"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.454007 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "b8666c84ba6f4bbba2eb4e712baa231f"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "c2d71eb50ea1ea34db21d841b14b6ec3"
server_key_iv: "5af8251886011809be5311ecff25ba15"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.458038 22019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:50:45.460552 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.461897 22019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:50:45.462057 22019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "b8666c84ba6f4bbba2eb4e712baa231f"
format_stamp: "Formatted at 2026-09-29 00:50:45 on dist-test-slave-5309"
server_key: "c2d71eb50ea1ea34db21d841b14b6ec3"
server_key_iv: "5af8251886011809be5311ecff25ba15"
server_key_version: "encryptionkey@0"
I20260929 00:50:45.462177 22019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:50:45.500303 22019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:50:45.503929 22019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:50:45.504170 22019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:50:45.504884 22019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:50:45.505800 22019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:50:45.505873 22019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.505946 22019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:50:45.505991 22019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:50:45.515690 22019 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:35733
I20260929 00:50:45.515717 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:35733 every 8 connection(s)
I20260929 00:50:45.516641 22019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:50:45.526463 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 22019
I20260929 00:50:45.526598 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
Sep 29 00:50:45 dist-test-slave-5309 krb5kdc[21644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:50:45.526899 11427 external_mini_cluster.cc:1428] Setting key e8fd349f248bc01ef10bf26b9b6144e9
I20260929 00:50:45.531505 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.518523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:56283 (local address 127.11.40.254:41293)
0929 00:50:45.518673 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.518677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.519592 (+ 915us) server_negotiation.cc:407] Connection header received
0929 00:50:45.521224 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.521228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.521289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.521401 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:45.522597 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.523169 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.524324 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.524554 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.524669 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.527264 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:45.527268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:45.527270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:45.527304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:45.528798 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.529824 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.529829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.529831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.529957 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.530334 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.530339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.530340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.530551 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:45.530666 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.531202 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.531338 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:50:45.532410 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.518947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:41293 (local address 127.11.40.195:56283)
0929 00:50:45.519416 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.519459 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.521005 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.521574 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.521581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.521964 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:50:45.522449 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.522462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.523324 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.523328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.524195 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.524204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.524716 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.524721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.524912 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.525533 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:50:45.525549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:50:45.527093 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:50:45.529183 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.529191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.529209 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.529694 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.530063 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:50:45.530067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:50:45.530069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:50:45.530229 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:50:45.530662 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:50:45.530669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:50:45.530948 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.531148 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.531522 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260929 00:50:45.533650 22148 heartbeater.cc:347] Connected to a master server at 127.11.40.254:41293
I20260929 00:50:45.533885 22148 heartbeater.cc:464] Registering TS with master...
I20260929 00:50:45.534349 22148 heartbeater.cc:511] Master 127.11.40.254:41293 requested a full tablet report, sending...
I20260929 00:50:45.535388 21689 ts_manager.cc:194] Registered new tserver with Master: b8666c84ba6f4bbba2eb4e712baa231f (127.11.40.195:35733)
I20260929 00:50:45.535975 21689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:56283
I20260929 00:50:45.542346 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:50:45.554296 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.544802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.11.40.254:41293)
0929 00:50:45.545066 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.545071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.545219 (+ 148us) server_negotiation.cc:407] Connection header received
0929 00:50:45.545462 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.545466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.545569 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.545669 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:50:45.546452 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.546994 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.547897 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.548125 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.548246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.550039 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:50:45.550043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:50:45.550045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:50:45.550075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:50:45.551856 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.552438 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.552442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.552444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.552542 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:50:45.553224 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:50:45.553230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:50:45.553235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:50:45.553530 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:50:45.553696 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.553965 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.554105 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260929 00:50:45.557969 21689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:50:45.561884 21689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:50:45.577876 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.572020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.11.40.193:43997)
0929 00:50:45.572708 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.572716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.572737 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:50:45.572826 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.572831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.572973 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.573607 (+ 634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:45.573769 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.574613 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.576318 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577138 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.577440 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.577608 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.577707 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":121,"threads_started":1}
I20260929 00:50:45.577869 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.571361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43997 (local address 127.0.0.1:45898)
0929 00:50:45.572196 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.572252 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.572406 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.573136 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.573143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.573169 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:45.573429 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.573438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.574734 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.574740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.576168 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.576179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577267 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.577395 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.577435 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.577558 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.577707 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":25,"spinlock_wait_cycles":5760,"thread_start_us":117,"threads_started":1}
I20260929 00:50:45.577939 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.571199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43493 (local address 127.0.0.1:53510)
0929 00:50:45.572178 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.572271 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.572491 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.573401 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.573404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.573425 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:45.573545 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.573550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.575147 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.575151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.576283 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.576292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577516 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.577609 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.577644 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.577712 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.577786 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":67,"threads_started":1}
I20260929 00:50:45.578320 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.571207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:35733 (local address 127.0.0.1:36802)
0929 00:50:45.572210 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.572253 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.572429 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.573778 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.573781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.573800 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:45.573893 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.573899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.575423 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.575427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.576621 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.576632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577915 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.578016 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.578038 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.578122 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.578188 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":734,"spinlock_wait_cycles":5248,"thread_start_us":105,"threads_started":1}
I20260929 00:50:45.578394 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.571317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.11.40.194:43493)
0929 00:50:45.571868 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.571881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.572357 (+ 476us) server_negotiation.cc:407] Connection header received
0929 00:50:45.572690 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.572697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.572951 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.573115 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:45.573772 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.574998 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.576411 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577186 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.577448 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.578131 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.578235 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":148,"threads_started":1}
I20260929 00:50:45.579591 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.571472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.11.40.195:35733)
0929 00:50:45.572933 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.572959 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.572986 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:50:45.573083 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.573088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.573326 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.573796 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:45.573991 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.575099 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.576758 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.577600 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.577713 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.577841 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.579374 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.579471 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":146,"threads_started":1}
I20260929 00:50:45.581724 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 9098d013b6f1473589723916a4abb432 (DEFAULT_TABLE table=test-table [id=7809f7bfc3d04115b227891070edf703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:45.581945 22082 tablet_service.cc:1467] Processing CreateTablet for tablet 9098d013b6f1473589723916a4abb432 (DEFAULT_TABLE table=test-table [id=7809f7bfc3d04115b227891070edf703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:45.582819 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098d013b6f1473589723916a4abb432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:45.582950 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098d013b6f1473589723916a4abb432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:45.585188 21948 tablet_service.cc:1467] Processing CreateTablet for tablet 9098d013b6f1473589723916a4abb432 (DEFAULT_TABLE table=test-table [id=7809f7bfc3d04115b227891070edf703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:50:45.586333 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098d013b6f1473589723916a4abb432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:50:45.589478 22165 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Bootstrap starting.
I20260929 00:50:45.591723 22165 tablet_bootstrap.cc:654] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:45.592499 22165 log.cc:824] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:45.593219 22166 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Bootstrap starting.
I20260929 00:50:45.593564 22168 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Bootstrap starting.
I20260929 00:50:45.594911 22165 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: No bootstrap required, opened a new log
I20260929 00:50:45.595153 22165 ts_tablet_manager.cc:1397] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260929 00:50:45.595731 22166 tablet_bootstrap.cc:654] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:45.597074 22168 tablet_bootstrap.cc:654] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Neither blocks nor log segments found. Creating new log.
I20260929 00:50:45.597251 22166 log.cc:824] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:45.597899 22168 log.cc:824] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Log is configured to *not* fsync() on all Append() calls
I20260929 00:50:45.599629 22166 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: No bootstrap required, opened a new log
I20260929 00:50:45.599531 22165 raft_consensus.cc:348] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.599856 22166 ts_tablet_manager.cc:1397] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:50:45.599854 22165 raft_consensus.cc:374] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:45.599927 22165 raft_consensus.cc:729] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dec35f1c4b942d48bffa4aedb9be62b, State: Initialized, Role: FOLLOWER
I20260929 00:50:45.600075 22168 tablet_bootstrap.cc:492] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: No bootstrap required, opened a new log
I20260929 00:50:45.600277 22168 ts_tablet_manager.cc:1397] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 00:50:45.600520 22165 consensus_queue.cc:260] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.601642 22165 ts_tablet_manager.cc:1428] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260929 00:50:45.602046 21880 heartbeater.cc:503] Master 127.11.40.254:41293 was elected leader, sending a full tablet report...
I20260929 00:50:45.603909 22166 raft_consensus.cc:348] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.604205 22166 raft_consensus.cc:374] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:45.604266 22166 raft_consensus.cc:729] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8666c84ba6f4bbba2eb4e712baa231f, State: Initialized, Role: FOLLOWER
I20260929 00:50:45.604749 22166 consensus_queue.cc:260] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.605572 22148 heartbeater.cc:503] Master 127.11.40.254:41293 was elected leader, sending a full tablet report...
I20260929 00:50:45.605434 22168 raft_consensus.cc:348] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.605693 22168 raft_consensus.cc:374] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:50:45.605741 22168 raft_consensus.cc:729] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae23aa372b5a419fb66ba83dd6b27eab, State: Initialized, Role: FOLLOWER
I20260929 00:50:45.605867 22166 ts_tablet_manager.cc:1428] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:50:45.606209 22168 consensus_queue.cc:260] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.607108 22014 heartbeater.cc:503] Master 127.11.40.254:41293 was elected leader, sending a full tablet report...
I20260929 00:50:45.607180 22168 ts_tablet_manager.cc:1428] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 00:50:45.757880 22015 tablet.cc:2369] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:45.767732 22149 tablet.cc:2369] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:50:45.770735 21881 tablet.cc:2369] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:50:45.945931 22172 raft_consensus.cc:482] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:50:45.946149 22172 raft_consensus.cc:504] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.947585 22172 leader_election.cc:288] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae23aa372b5a419fb66ba83dd6b27eab (127.11.40.194:43493), 8dec35f1c4b942d48bffa4aedb9be62b (127.11.40.193:43997)
I20260929 00:50:45.951507 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.948030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:50337 (local address 127.11.40.193:43997)
0929 00:50:45.948161 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.948167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.948182 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:50:45.948314 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.948316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.948363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.948470 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:45.948757 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.949360 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.950454 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.950910 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.951050 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.951106 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.951290 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.951372 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:50:45.951507 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.947898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43997 (local address 127.11.40.195:50337)
0929 00:50:45.948059 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.948077 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.948194 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.948484 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.948487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.948505 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:45.948633 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.948638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.949486 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.949491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.950316 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.950325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.951044 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.951048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.951157 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.951182 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.951273 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.951347 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:50:45.952183 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.947919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43493 (local address 127.11.40.195:41415)
0929 00:50:45.948666 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:45.948678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:50:45.948808 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:50:45.949170 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:50:45.949173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:50:45.949190 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:50:45.949331 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.949337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.950104 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.950111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.950929 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:50:45.950939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.951768 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.951772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:50:45.951875 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.951902 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:50:45.951994 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:50:45.952065 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":353,"thread_start_us":160,"threads_started":1}
I20260929 00:50:45.952235 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:45.948004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:41415 (local address 127.11.40.194:43493)
0929 00:50:45.948129 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:45.948134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:45.948780 (+ 646us) server_negotiation.cc:407] Connection header received
0929 00:50:45.948988 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:45.948991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:45.949048 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:45.949151 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:50:45.949457 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.949975 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.951071 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:45.951476 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:45.951636 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:45.951721 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:45.952043 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 00:50:45.952124 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:50:45.952343 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098d013b6f1473589723916a4abb432" candidate_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" is_pre_election: true
I20260929 00:50:45.952734 21834 raft_consensus.cc:2454] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8666c84ba6f4bbba2eb4e712baa231f in term 0.
I20260929 00:50:45.953054 21968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098d013b6f1473589723916a4abb432" candidate_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" is_pre_election: true
I20260929 00:50:45.953384 21968 raft_consensus.cc:2454] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8666c84ba6f4bbba2eb4e712baa231f in term 0.
I20260929 00:50:45.953354 22035 leader_election.cc:302] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dec35f1c4b942d48bffa4aedb9be62b, b8666c84ba6f4bbba2eb4e712baa231f; no voters:
I20260929 00:50:45.953670 22172 raft_consensus.cc:2781] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:50:45.953784 22172 raft_consensus.cc:482] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:50:45.953852 22172 raft_consensus.cc:3037] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:45.954890 22172 raft_consensus.cc:504] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.955279 22172 leader_election.cc:288] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [CANDIDATE]: Term 1 election: Requested vote from peers ae23aa372b5a419fb66ba83dd6b27eab (127.11.40.194:43493), 8dec35f1c4b942d48bffa4aedb9be62b (127.11.40.193:43997)
I20260929 00:50:45.955662 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098d013b6f1473589723916a4abb432" candidate_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dec35f1c4b942d48bffa4aedb9be62b"
I20260929 00:50:45.955672 21968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098d013b6f1473589723916a4abb432" candidate_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae23aa372b5a419fb66ba83dd6b27eab"
I20260929 00:50:45.955806 21834 raft_consensus.cc:3037] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:45.955821 21968 raft_consensus.cc:3037] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:50:45.956911 21968 raft_consensus.cc:2454] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8666c84ba6f4bbba2eb4e712baa231f in term 1.
I20260929 00:50:45.957012 21834 raft_consensus.cc:2454] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8666c84ba6f4bbba2eb4e712baa231f in term 1.
I20260929 00:50:45.957238 22038 leader_election.cc:302] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae23aa372b5a419fb66ba83dd6b27eab, b8666c84ba6f4bbba2eb4e712baa231f; no voters:
I20260929 00:50:45.957410 22172 raft_consensus.cc:2781] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:50:45.957624 22172 raft_consensus.cc:686] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 1 LEADER]: Becoming Leader. State: Replica: b8666c84ba6f4bbba2eb4e712baa231f, State: Running, Role: LEADER
I20260929 00:50:45.957967 22172 consensus_queue.cc:237] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } }
I20260929 00:50:45.961393 21689 catalog_manager.cc:5495] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f reported cstate change: term changed from 0 to 1, leader changed from <none> to b8666c84ba6f4bbba2eb4e712baa231f (127.11.40.195). New cstate: current_term: 1 leader_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8666c84ba6f4bbba2eb4e712baa231f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 } health_report { overall_health: UNKNOWN } } }
I20260929 00:50:46.032318 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:46.028651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.11.40.195:35733)
0929 00:50:46.028783 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:50:46.028786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:50:46.028833 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:50:46.028968 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:50:46.028970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:50:46.029033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:50:46.029119 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:50:46.029409 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:46.029928 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:46.030822 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:50:46.031044 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:50:46.031120 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:50:46.031492 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:50:46.031923 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:50:46.032014 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:50:46.032075 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:50:46.032128 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:50:46.038637 21968 raft_consensus.cc:1261] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 1 FOLLOWER]: Refusing update from remote peer b8666c84ba6f4bbba2eb4e712baa231f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:46.038637 21834 raft_consensus.cc:1261] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 1 FOLLOWER]: Refusing update from remote peer b8666c84ba6f4bbba2eb4e712baa231f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:50:46.039489 22172 consensus_queue.cc:1035] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dec35f1c4b942d48bffa4aedb9be62b" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:46.039723 22175 consensus_queue.cc:1035] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae23aa372b5a419fb66ba83dd6b27eab" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:50:46.049136 22179 mvcc.cc:205] Tried to move back new op lower bound from 7334473916566159360 to 7334473916246581248. Current Snapshot: MvccSnapshot[applied={T|T < 7334473916566159360}]
I20260929 00:50:46.049342 22180 mvcc.cc:205] Tried to move back new op lower bound from 7334473916566159360 to 7334473916246581248. Current Snapshot: MvccSnapshot[applied={T|T < 7334473916566159360}]
I20260929 00:50:46.056744 21688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:46.057004 21688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:50:46.059473 21688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3516e37f27d4f3e97d9ca7171ff3f84: Sending DeleteTablet for 3 replicas of tablet 9098d013b6f1473589723916a4abb432
I20260929 00:50:46.060300 21948 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098d013b6f1473589723916a4abb432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:46 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:53510
I20260929 00:50:46.060304 22082 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098d013b6f1473589723916a4abb432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:46 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:36802
I20260929 00:50:46.060846 22188 tablet_replica.cc:330] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: stopping tablet replica
I20260929 00:50:46.060871 21814 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098d013b6f1473589723916a4abb432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:50:46 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:45898
I20260929 00:50:46.061177 22188 raft_consensus.cc:2229] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:50:46.061390 22190 tablet_replica.cc:330] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b: stopping tablet replica
I20260929 00:50:46.061582 22188 raft_consensus.cc:2258] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:46.061846 22190 raft_consensus.cc:2229] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:46.061980 22189 tablet_replica.cc:330] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: stopping tablet replica
I20260929 00:50:46.062294 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21751
I20260929 00:50:46.062275 22189 raft_consensus.cc:2229] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:50:46.062291 22190 raft_consensus.cc:2258] T 9098d013b6f1473589723916a4abb432 P 8dec35f1c4b942d48bffa4aedb9be62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:46.062587 22189 raft_consensus.cc:2258] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:50:46.063169 22188 ts_tablet_manager.cc:1905] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:46.066556 22188 ts_tablet_manager.cc:1918] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:46.066671 22188 log.cc:1196] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/9098d013b6f1473589723916a4abb432
I20260929 00:50:46.067008 22188 ts_tablet_manager.cc:1939] T 9098d013b6f1473589723916a4abb432 P b8666c84ba6f4bbba2eb4e712baa231f: Deleting consensus metadata
I20260929 00:50:46.068197 22189 ts_tablet_manager.cc:1905] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:50:46.070971 22189 ts_tablet_manager.cc:1918] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:50:46.071071 22189 log.cc:1196] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/9098d013b6f1473589723916a4abb432
I20260929 00:50:46.071246 21675 catalog_manager.cc:4841] TS b8666c84ba6f4bbba2eb4e712baa231f (127.11.40.195:35733): tablet 9098d013b6f1473589723916a4abb432 (table test-table [id=7809f7bfc3d04115b227891070edf703]) successfully deleted
I20260929 00:50:46.071396 22189 ts_tablet_manager.cc:1939] T 9098d013b6f1473589723916a4abb432 P ae23aa372b5a419fb66ba83dd6b27eab: Deleting consensus metadata
W20260929 00:50:46.071596 21673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:43997 (error 108)
I20260929 00:50:46.072544 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21885
I20260929 00:50:46.072595 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:50:46.072295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:43997 (local address 127.0.0.1:45904)
0929 00:50:46.072429 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 00:50:46.072505 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 00:50:46.072628 21676 catalog_manager.cc:4841] TS ae23aa372b5a419fb66ba83dd6b27eab (127.11.40.194:43493): tablet 9098d013b6f1473589723916a4abb432 (table test-table [id=7809f7bfc3d04115b227891070edf703]) successfully deleted
W20260929 00:50:46.073921 21673 catalog_manager.cc:4584] TS 8dec35f1c4b942d48bffa4aedb9be62b (127.11.40.193:43997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9098d013b6f1473589723916a4abb432: Network error: Client connection negotiation failed: client connection to 127.11.40.193:43997: connect: Connection refused (error 111)
I20260929 00:50:46.078680 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 22019
I20260929 00:50:46.084213 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 21657
2026-09-29T00:50:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2762 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:50:46 dist-test-slave-5309 krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:50:46 dist-test-slave-5309 krb5kdc[22194](info): set up 2 sockets
Sep 29 00:50:46 dist-test-slave-5309 krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260929 00:50:47.135269 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:50:47 dist-test-slave-5309 krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:50:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/rangeradmin_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/rangeradmin_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/rangerlookup_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/rangerlookup_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
I20260929 00:50:47.312599 11427 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJXuMFZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/postgres -l logfile start
2026-09-29 00:50:50.151 UTC [22224] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:50:50.151 UTC [22224] LOG: listening on IPv4 address "127.11.40.212", port 51035
2026-09-29 00:50:50.155 UTC [22224] LOG: listening on Unix socket "/tmp/.s.PGSQL.51035"
2026-09-29 00:50:50.176 UTC [22227] LOG: database system was shut down at 2026-09-29 00:50:48 UTC
2026-09-29 00:50:50.180 UTC [22224] LOG: database system is ready to accept connections
I20260929 00:50:51.143015 11427 mini_postgres.cc:96] Postgres bound to 51035
2026-09-29 00:50:51.150 UTC [22235] FATAL: database "slave" does not exist
127.11.40.212:51035 - accepting connections
I20260929 00:50:51.151085 11427 mini_ranger.cc:161] Starting Ranger...
I20260929 00:50:51.169253 11427 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:50:51.741422 11427 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:50:51.741533 11427 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin
2026-09-29 00:50:52,227 [I] DB FLAVOR :POSTGRES
2026-09-29 00:50:52,228 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:50:52,228 [I] Checking connection..
2026-09-29 00:50:52,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:50:52,551 [I] Checking connection passed.
2026-09-29 00:50:52,551 [I] --------- Verifying version history table ---------
2026-09-29 00:50:52,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:50:52,819 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:50:52,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:50:53,067 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:50:53,067 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:50:53,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:50:53.323 UTC [22299] WARNING: there is no transaction in progress
2026-09-29 00:50:53,329 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:50:53,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:50:53,595 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:50:53,596 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:50:53,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:50:53,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:50:54,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5309.c.gcp-upstream.internal','N') ;"
2026-09-29 00:50:54,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:50:54,567 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:50:54,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:50:54,832 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:50:54,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:50:55,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:50:55,355 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:50:55,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:50:56.478 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.497 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.518 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.539 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.565 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.589 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.659 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.668 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.688 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.699 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.710 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.726 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.737 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.746 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.754 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:56.763 UTC [22407] WARNING: there is no transaction in progress
2026-09-29 00:50:57,534 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:50:57,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:50:57,786 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:50:57,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:50:58,051 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:50:58,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:50:58,294 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:50:58,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:50:58,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:50:58,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5309.c.gcp-upstream.internal';"
2026-09-29 00:50:59,059 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:50:59,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:51035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:50:59,310 [I] DB_PATCHES have already been applied
I20260929 00:50:59.315789 11427 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJXuMFZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:50:59.315861 11427 mini_ranger.cc:193] Using host: 127.11.40.212
I20260929 00:50:59.319585 11427 mini_ranger.cc:227] Ranger admin URL: http://127.11.40.212:59819
Sep 29, 2026 12:50:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:50:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:50:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:50:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:50:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:51:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:51:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 29, 2026 12:51:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:51:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 12:51:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:51:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:51:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:53:00.357952 11427 mini_ranger.cc:160] Time spent starting Ranger: real 129.207s user 0.017s sys 0.158s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 22494 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p22494
f310
n127.0.0.1:54772->127.11.40.212:51035
f311
n127.0.0.1:54754->127.11.40.212:51035
f312
n127.0.0.1:54756->127.11.40.212:51035
f313
n127.0.0.1:54778->127.11.40.212:51035
f314
n127.0.0.1:54790->127.11.40.212:51035'
I20260929 00:53:00.359262 11427 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:53:00.390201 11427 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:53:00.390 UTC [22224] LOG: received smart shutdown request
2026-09-29 00:53:00.394 UTC [22224] LOG: background worker "logical replication launcher" (PID 22233) exited with exit code 1
2026-09-29 00:53:00.395 UTC [22228] LOG: shutting down
2026-09-29 00:53:00.623 UTC [22224] LOG: database system is shut down
2026-09-29T00:53:00Z chronyd exiting
I20260929 00:53:00.645648 11427 test_util.cc:175] -----------------------------------------------
I20260929 00:53:00.645713 11427 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790642980693194-11427-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (134540 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:53:00 dist-test-slave-5309 krb5kdc[22614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:53:00 dist-test-slave-5309 krb5kdc[22614](info): set up 2 sockets
Sep 29 00:53:00 dist-test-slave-5309 krb5kdc[22614](info): commencing operation
krb5kdc: starting...
W20260929 00:53:01.694198 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:53:01 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:53:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangeradmin_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangeradmin_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangerlookup_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangerlookup_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
I20260929 00:53:01.913240 11427 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJXuMFZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/postgres -l logfile start
2026-09-29 00:53:05.056 UTC [22644] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:53:05.057 UTC [22644] LOG: listening on IPv4 address "127.11.40.212", port 57173
2026-09-29 00:53:05.061 UTC [22644] LOG: listening on Unix socket "/tmp/.s.PGSQL.57173"
2026-09-29 00:53:05.084 UTC [22647] LOG: database system was shut down at 2026-09-29 00:53:03 UTC
2026-09-29 00:53:05.089 UTC [22644] LOG: database system is ready to accept connections
I20260929 00:53:06.049772 11427 mini_postgres.cc:96] Postgres bound to 57173
2026-09-29 00:53:06.056 UTC [22655] FATAL: database "slave" does not exist
127.11.40.212:57173 - accepting connections
I20260929 00:53:06.057101 11427 mini_ranger.cc:161] Starting Ranger...
I20260929 00:53:06.073603 11427 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:53:06.754601 11427 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:53:06.754719 11427 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin
2026-09-29 00:53:07,057 [I] DB FLAVOR :POSTGRES
2026-09-29 00:53:07,058 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:53:07,058 [I] Checking connection..
2026-09-29 00:53:07,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:53:07,315 [I] Checking connection passed.
2026-09-29 00:53:07,315 [I] --------- Verifying version history table ---------
2026-09-29 00:53:07,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:53:07,573 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:53:07,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:53:07,847 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:53:07,847 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:53:07,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:53:08.128 UTC [22719] WARNING: there is no transaction in progress
2026-09-29 00:53:08,135 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:53:08,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:53:08,394 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:53:08,394 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:53:08,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:53:08,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:53:08,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5309.c.gcp-upstream.internal','N') ;"
2026-09-29 00:53:09,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:53:09,439 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:53:09,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:53:09,705 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:53:09,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:53:09,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:53:10,263 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:53:10,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:53:11.366 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.384 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.406 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.425 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.444 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.466 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.537 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.546 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.564 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.575 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.588 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.604 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.615 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.624 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.633 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:11.642 UTC [22827] WARNING: there is no transaction in progress
2026-09-29 00:53:12,527 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:53:12,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:53:12,756 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:53:12,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:53:13,027 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:53:13,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:53:13,321 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:53:13,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:53:13,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:53:13,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5309.c.gcp-upstream.internal';"
2026-09-29 00:53:14,113 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:53:14,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:53:14,386 [I] DB_PATCHES have already been applied
I20260929 00:53:14.392763 11427 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJXuMFZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:53:14.392894 11427 mini_ranger.cc:193] Using host: 127.11.40.212
I20260929 00:53:14.396476 11427 mini_ranger.cc:227] Ranger admin URL: http://127.11.40.212:59097
Sep 29, 2026 12:53:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:53:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:53:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:53:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:53:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:53:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:53:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 00:53:15.584203 11427 mini_ranger.cc:160] Time spent starting Ranger: real 9.527s user 0.001s sys 0.009s
Sep 29, 2026 12:53:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:53:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 12:53:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:53:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:53:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:53:37.954833 11427 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangerkms_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/rangerkms_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.40.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.40.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.40.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/HTTP_127.11.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 00:53:38 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643218, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 00:53:38.057204 11427 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 00:53:38.070433 11427 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 00:53:39.174172 11427 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 00:53:39.174278 11427 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms
I20260929 00:53:39.174377 11427 mini_ranger_kms.cc:227] Using postgres at 127.11.40.212:57173
2026-09-29 00:53:39,807 [I] DB FLAVOR :POSTGRES
2026-09-29 00:53:39,808 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:53:39,808 [I] Checking connection
2026-09-29 00:53:39,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 00:53:40,062 [I] connection success
2026-09-29 00:53:40,062 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 00:53:40,062 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 00:53:40,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 00:53:40,337 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 00:53:40,337 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:53:40,338 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 00:53:40,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.40.212:57173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 00:53:40,605 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 00:53:41.164729 11427 mini_ranger_kms.cc:310] Created kms service
I20260929 00:53:41.354358 11427 mini_ranger_kms.cc:326] Created kudu user
I20260929 00:53:41.404297 11427 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 00:53:41 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643218, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:41.905741 11427 mini_ranger_kms.cc:384] Added ranger policy
I20260929 00:53:41.905907 11427 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJXuMFZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJXuMFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 00:53:41.905927 11427 mini_ranger_kms.cc:240] Using host: 127.11.40.212
I20260929 00:53:41.910331 11427 mini_ranger_kms.cc:276] Ranger KMS PID: 23008
I20260929 00:53:41.910492 11427 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.40.212:50397
Sep 29, 2026 12:53:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:53:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:53:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 12:53:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 12:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 12:53:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 00:53:44 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.40.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:44.235137 11427 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.178s user 0.039s sys 0.324s
I20260929 00:53:44.235280 11427 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 00:53:44.235385 11427 mini_ranger_kms.cc:401] 127.11.40.212:50397/kms/v1/keys
Sep 29 00:53:44 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.40.212@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
Sep 29 00:53:45 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.40.212@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
Sep 29 00:53:47 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:53:47.560761 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35239
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:35239
--ranger_config_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:47.690512 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:47.690820 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:47.690860 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:47.694936 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:47.695005 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:47.695025 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:47.695042 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:47.695057 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:47.700677 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.40.212:50397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:35239
--ranger_config_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:47.701814 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:47.702948 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:47.714234 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:47.714190 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:47.714236 23058 server_base.cc:1034] running on GCE node
W20260929 00:53:47.714352 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:47.715096 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:47.716418 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:47.717640 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790643227717610 us; error 37 us; skew 500 ppm
Sep 29 00:53:47 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643227, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:47.725908 23058 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:53:47.726964 23058 webserver.cc:460] Webserver started at http://127.11.40.254:37265/ using document root <none> and password file <none>
I20260929 00:53:47.727571 23058 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:47.727656 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:47.727932 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:53:47 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643227, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:47.777490 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "c467f9054a0d4f158e148283e5c53ceb"
format_stamp: "Formatted at 2026-09-29 00:53:47 on dist-test-slave-5309"
server_key: "72b1f7ece365e83f83322b24ff530b23"
server_key_iv: "d3b91ca03a2940860c5d4a65e5c173b3"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:47.778301 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "c467f9054a0d4f158e148283e5c53ceb"
format_stamp: "Formatted at 2026-09-29 00:53:47 on dist-test-slave-5309"
server_key: "72b1f7ece365e83f83322b24ff530b23"
server_key_iv: "d3b91ca03a2940860c5d4a65e5c173b3"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:47.783207 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:53:47.803112 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:47.804930 23058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:53:47.805145 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "c467f9054a0d4f158e148283e5c53ceb"
format_stamp: "Formatted at 2026-09-29 00:53:47 on dist-test-slave-5309"
server_key: "72b1f7ece365e83f83322b24ff530b23"
server_key_iv: "d3b91ca03a2940860c5d4a65e5c173b3"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:47.805323 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:47.828523 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:47.832172 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:47.832383 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:47.841645 23058 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:35239
I20260929 00:53:47.841683 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:35239 every 8 connection(s)
I20260929 00:53:47.842877 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:53:47.845810 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23058
I20260929 00:53:47.845958 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:53:47.846302 11427 external_mini_cluster.cc:1428] Setting key 589bddc6c94fc215a918010ed5792109
I20260929 00:53:47.846573 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:47.854355 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Bootstrap starting.
Sep 29 00:53:47 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:47.857734 23126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:47.858664 23126 log.cc:824] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:47.861451 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: No bootstrap required, opened a new log
I20260929 00:53:47.864466 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:47.847844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.11.40.254:35239)
0929 00:53:47.848356 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:47.848377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:47.848425 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:53:47.849356 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:47.849380 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:47.849828 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:47.850211 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:47.851166 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:47.852764 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:47.853999 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:47.854718 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:47.854887 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:47.858240 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:47.858247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:47.858264 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:47.858299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:47.861112 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:47.861653 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:47.861659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:47.861666 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:47.861800 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:47.862132 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:47.862144 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:47.862146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:47.862685 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:47.862862 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:47.863206 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 00:53:47.863490 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":149,"threads_started":1}
I20260929 00:53:47.865694 23126 raft_consensus.cc:348] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:47.865989 23126 raft_consensus.cc:374] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:47.866074 23126 raft_consensus.cc:729] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c467f9054a0d4f158e148283e5c53ceb, State: Initialized, Role: FOLLOWER
I20260929 00:53:47.866598 23126 consensus_queue.cc:260] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:47.866765 23126 raft_consensus.cc:388] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:53:47.866846 23126 raft_consensus.cc:482] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:53:47.866951 23126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:47.868073 23126 raft_consensus.cc:504] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:47.868450 23126 leader_election.cc:302] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c467f9054a0d4f158e148283e5c53ceb; no voters:
I20260929 00:53:47.868875 23126 leader_election.cc:288] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:53:47.868947 23131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:53:47.869163 23131 raft_consensus.cc:686] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 LEADER]: Becoming Leader. State: Replica: c467f9054a0d4f158e148283e5c53ceb, State: Running, Role: LEADER
I20260929 00:53:47.869520 23131 consensus_queue.cc:237] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:47.870306 23126 sys_catalog.cc:564] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:47.870803 23131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c467f9054a0d4f158e148283e5c53ceb. Latest consensus state: current_term: 1 leader_uuid: "c467f9054a0d4f158e148283e5c53ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } } }
I20260929 00:53:47.871753 23131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:47.871640 23132 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c467f9054a0d4f158e148283e5c53ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } } }
I20260929 00:53:47.872175 23132 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:47.872726 23126 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:53:47.874125 23141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:53:47.880060 23141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:53:47.887745 23141 catalog_manager.cc:1347] Generated new cluster ID: 7ef8c70776144cc9b86ea17ac2341527
I20260929 00:53:47.887821 23141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:53:47.897792 23141 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:53:47.899147 23141 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:53:47.905469 23141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Generated new TSK 0
I20260929 00:53:47.906426 23141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:53:48.665233 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:48.786226 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:48.786527 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:48.786669 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:48.791551 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:48.791644 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:48.791750 23172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:53:48.797426 23172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:48.798645 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:48.799747 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:48.811287 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:48.811408 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:48.811926 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:48.812795 23172 server_base.cc:1034] running on GCE node
I20260929 00:53:48.813427 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:48.814337 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:48.817658 23172 hybrid_clock.cc:648] HybridClock initialized: now 1790643228817563 us; error 99 us; skew 500 ppm
Sep 29 00:53:48 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:48.821074 23172 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:53:48.822352 23172 webserver.cc:460] Webserver started at http://127.11.40.193:43023/ using document root <none> and password file <none>
I20260929 00:53:48.823072 23172 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:48.823215 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:48.823495 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:53:48 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:48.844579 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "9d114bf0c0634fc6b6b652652d3e1acb"
format_stamp: "Formatted at 2026-09-29 00:53:48 on dist-test-slave-5309"
server_key: "42a05b91d98fe9d724c173ff4876a344"
server_key_iv: "55a034656ae59943ca6faddf950bdded"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:48.845340 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "9d114bf0c0634fc6b6b652652d3e1acb"
format_stamp: "Formatted at 2026-09-29 00:53:48 on dist-test-slave-5309"
server_key: "42a05b91d98fe9d724c173ff4876a344"
server_key_iv: "55a034656ae59943ca6faddf950bdded"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:48.850369 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:53:48.881271 23189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:48.883278 23172 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260929 00:53:48.883431 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "9d114bf0c0634fc6b6b652652d3e1acb"
format_stamp: "Formatted at 2026-09-29 00:53:48 on dist-test-slave-5309"
server_key: "42a05b91d98fe9d724c173ff4876a344"
server_key_iv: "55a034656ae59943ca6faddf950bdded"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:48.883572 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:48.916136 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:48.920284 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:48.920572 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:48.921482 23172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:48.922755 23172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:48.922879 23172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 00:53:48.923019 23172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:48.923102 23172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:48.935667 23172 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:37327
I20260929 00:53:48.935693 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:37327 every 8 connection(s)
I20260929 00:53:48.936659 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:53:48.946168 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23172
I20260929 00:53:48.946319 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:53:48.946707 11427 external_mini_cluster.cc:1428] Setting key 688a71bbf3a5c3fd0eeb59d5625c896e
Sep 29 00:53:48 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:48.955263 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:48.939730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.193:34467)
0929 00:53:48.940314 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:48.940362 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:48.941337 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:48.941987 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:48.941995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:48.942295 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:48.942892 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:48.942909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:48.943786 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:48.943792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:48.944711 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:48.944720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:48.945333 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:48.945337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:48.945542 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:48.946247 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:48.946276 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:48.948331 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:48.951262 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:48.951269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:48.951284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:48.951600 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:48.951999 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:48.952003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:48.952006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:48.952259 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:48.952925 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:48.952938 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:48.953343 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 00:53:48.953578 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 00:53:48.953976 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":144,"threads_started":1}
I20260929 00:53:48.957070 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:48.939248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:34467 (local address 127.11.40.254:35239)
0929 00:53:48.939483 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:48.939487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:48.940499 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 00:53:48.941606 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:48.941610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:48.941688 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:48.941804 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:48.943050 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:48.943640 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:48.944908 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:48.945151 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:48.945273 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:48.948543 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:48.948548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:48.948550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:48.948576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:48.951067 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:48.951726 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:48.951733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:48.951737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:48.951861 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:48.952425 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:48.952430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:48.952434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:48.952731 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:48.956434 (+ 3703us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:48.956549 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 00:53:48.956741 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260929 00:53:48.958535 23304 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:48.958792 23304 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:48.959295 23304 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:48.961369 23090 ts_manager.cc:194] Registered new tserver with Master: 9d114bf0c0634fc6b6b652652d3e1acb (127.11.40.193:37327)
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:53:48.962780 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:34467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:53:48.988651 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 00:53:49.159447 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:49.160002 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:49.160092 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:49.166970 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:49.167114 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:49.167227 23311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:53:49.175602 23311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:49.177174 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:49.178709 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:49.191555 23320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:49.191572 23317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:49.191757 23318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:49.192582 23311 server_base.cc:1034] running on GCE node
I20260929 00:53:49.193276 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:49.194139 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:49.195355 23311 hybrid_clock.cc:648] HybridClock initialized: now 1790643229195333 us; error 39 us; skew 500 ppm
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:49.199635 23311 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:53:49.200955 23311 webserver.cc:460] Webserver started at http://127.11.40.194:44135/ using document root <none> and password file <none>
I20260929 00:53:49.201781 23311 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:49.201886 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:49.202404 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:49.218854 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "78932ab64c254dbda140735f343eb297"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "1eea4e562d9951ff3a3482879fa9fa9c"
server_key_iv: "7b9a87288f9378329a67f0a918684711"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.219894 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "78932ab64c254dbda140735f343eb297"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "1eea4e562d9951ff3a3482879fa9fa9c"
server_key_iv: "7b9a87288f9378329a67f0a918684711"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.231782 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.004s
I20260929 00:53:49.250603 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.252382 23311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:53:49.252524 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "78932ab64c254dbda140735f343eb297"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "1eea4e562d9951ff3a3482879fa9fa9c"
server_key_iv: "7b9a87288f9378329a67f0a918684711"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.252667 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:49.267786 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:49.279125 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:49.279439 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:49.280262 23311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:49.282768 23311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:49.282878 23311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.282995 23311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:49.283061 23311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.300649 23311 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:39689
I20260929 00:53:49.302136 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:53:49.303440 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23311
I20260929 00:53:49.303563 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:53:49.303861 11427 external_mini_cluster.cc:1428] Setting key 34c0647c07b37bd5101ea8adb583d0b6
I20260929 00:53:49.304293 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:39689 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:49.332098 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:49.314763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.194:49229)
0929 00:53:49.315780 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:49.315843 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:49.316930 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:49.317758 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:49.317769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:49.318373 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:49.319315 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:49.319342 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.320552 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.320555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:49.321714 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:49.321725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.322611 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.322616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:49.322869 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:49.323618 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:49.323644 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:49.325646 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:49.328256 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:49.328266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:49.328292 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:49.328736 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:49.329126 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:49.329130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:49.329134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:49.329344 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:49.329920 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:49.329930 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:49.330317 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 00:53:49.330552 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 00:53:49.330982 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":181,"threads_started":1}
I20260929 00:53:49.333060 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:49.314080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:49229 (local address 127.11.40.254:35239)
0929 00:53:49.314236 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:49.314252 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:49.315979 (+ 1727us) server_negotiation.cc:407] Connection header received
0929 00:53:49.317267 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:49.317270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:49.317332 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:49.317451 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:49.319536 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.320400 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.321868 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.322281 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.322398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:49.325936 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:49.325943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:49.325950 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:49.325988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:49.328074 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:49.328899 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:49.328903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:49.328905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:49.329003 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:49.329471 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:49.329476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:49.329479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:49.329756 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:49.329861 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:49.332474 (+ 2613us) server_negotiation.cc:299] Negotiation successful
0929 00:53:49.332881 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:53:49.334198 23441 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:49.334462 23441 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:49.335161 23441 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:49.336462 23090 ts_manager.cc:194] Registered new tserver with Master: 78932ab64c254dbda140735f343eb297 (127.11.40.194:39689)
I20260929 00:53:49.337258 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:49229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:53:49.341874 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:49.511461 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:49.512154 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:49.512288 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:49.518220 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:49.518419 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:49.518558 23446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:53:49.526788 23446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:49.528467 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:49.529999 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:49.560614 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:49.561075 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:49.563210 23446 server_base.cc:1034] running on GCE node
W20260929 00:53:49.564993 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:49.565635 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:49.566520 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:49.573944 23446 hybrid_clock.cc:648] HybridClock initialized: now 1790643229573848 us; error 116 us; skew 500 ppm
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:49.577657 23446 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:53:49.578899 23446 webserver.cc:460] Webserver started at http://127.11.40.195:44077/ using document root <none> and password file <none>
I20260929 00:53:49.579651 23446 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:49.579803 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:49.580078 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:49.601315 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "71ed85404e6f4ea6866de0e15b9370e2"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "770d0c4cec39caf5f4f7a5d81283f2f2"
server_key_iv: "8e6b7d68a7baa1e51c30cb3b82ca064c"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.602309 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "71ed85404e6f4ea6866de0e15b9370e2"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "770d0c4cec39caf5f4f7a5d81283f2f2"
server_key_iv: "8e6b7d68a7baa1e51c30cb3b82ca064c"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.608764 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 00:53:49.645318 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.647396 23446 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.000s
I20260929 00:53:49.647550 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "71ed85404e6f4ea6866de0e15b9370e2"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "770d0c4cec39caf5f4f7a5d81283f2f2"
server_key_iv: "8e6b7d68a7baa1e51c30cb3b82ca064c"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.647708 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:49.679715 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:49.683929 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:49.684136 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:49.684904 23446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:49.685985 23446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:49.686051 23446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.686105 23446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:49.686174 23446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 00:53:49.698249 23446 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:44073
I20260929 00:53:49.698485 23574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:44073 every 8 connection(s)
I20260929 00:53:49.699656 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:53:49.703992 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23446
I20260929 00:53:49.704107 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:53:49.704421 11427 external_mini_cluster.cc:1428] Setting key 5d272666c613e0dfdedd8ff238a9d8d8
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:49.727977 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:49.702166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:54611 (local address 127.11.40.254:35239)
0929 00:53:49.702296 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:49.702300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:49.705296 (+ 2996us) server_negotiation.cc:407] Connection header received
0929 00:53:49.706502 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:49.706505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:49.706559 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:49.706611 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:49.708390 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.709327 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.711080 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.711726 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.711951 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:49.715390 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:49.715394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:49.715397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:49.715427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:49.724956 (+ 9529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:49.725949 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:49.725953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:49.725955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:49.726070 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:49.726541 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:49.726544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:49.726546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:49.726788 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:49.726872 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:49.727650 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 00:53:49.727808 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:53:49.731539 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:49.702718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.195:54611)
0929 00:53:49.705128 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:49.705189 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:49.706253 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:49.707141 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:49.707151 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:49.707510 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:49.708202 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:49.708222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.709483 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.709487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:49.710924 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:49.710935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:49.712010 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:49.712015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:49.712255 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:49.713189 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:49.713210 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:49.715143 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:49.725197 (+ 10054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:49.725207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:49.725230 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:49.725782 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:49.726188 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:49.726191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:49.726193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:49.726429 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:49.726963 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:49.726971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:49.727353 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 00:53:49.728547 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0929 00:53:49.728999 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2166,"thread_start_us":146,"threads_started":1}
I20260929 00:53:49.733170 23575 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:49.733443 23575 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:49.734048 23575 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:49.735333 23090 ts_manager.cc:194] Registered new tserver with Master: 71ed85404e6f4ea6866de0e15b9370e2 (127.11.40.195:44073)
I20260929 00:53:49.736096 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:54611
I20260929 00:53:49.742722 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:53:49.743484 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23172
I20260929 00:53:49.752007 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23311
I20260929 00:53:49.759398 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23446
I20260929 00:53:49.766752 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23058
I20260929 00:53:49.774804 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35239
--webserver_interface=127.11.40.254
--webserver_port=37265
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:35239
--ranger_config_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:49.893103 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:49.893393 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:49.893433 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:49.897660 23578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:49.897735 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:49.897755 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:49.897789 23578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:49.897804 23578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:49.903394 23578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:35239
--ranger_config_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=37265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:49.904533 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:49.905915 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:49.915293 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:49.915388 23578 server_base.cc:1034] running on GCE node
W20260929 00:53:49.915323 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:49.915313 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:49.916167 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:49.917410 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:49.918641 23578 hybrid_clock.cc:648] HybridClock initialized: now 1790643229918583 us; error 49 us; skew 500 ppm
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:49.921516 23578 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:53:49.922401 23578 webserver.cc:460] Webserver started at http://127.11.40.254:37265/ using document root <none> and password file <none>
I20260929 00:53:49.922900 23578 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:49.922966 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:53:49 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643229, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:49.986382 23578 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:53:49.988736 23594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:49.990087 23578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:53:49.990198 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "c467f9054a0d4f158e148283e5c53ceb"
format_stamp: "Formatted at 2026-09-29 00:53:47 on dist-test-slave-5309"
server_key: "72b1f7ece365e83f83322b24ff530b23"
server_key_iv: "d3b91ca03a2940860c5d4a65e5c173b3"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:49.990743 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:50.004215 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:50.007586 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:50.008002 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:50.016466 23646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:35239 every 8 connection(s)
I20260929 00:53:50.016463 23578 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:35239
I20260929 00:53:50.017673 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:53:50.022042 23647 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:53:50.023644 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23578
I20260929 00:53:50.023882 23647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Bootstrap starting.
I20260929 00:53:50.024147 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:37327
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:35239
--webserver_port=43023
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:53:50.041467 23647 log.cc:824] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:50.048134 23647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:53:50.048583 23647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Bootstrap complete.
I20260929 00:53:50.052772 23647 raft_consensus.cc:348] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:50.053226 23647 raft_consensus.cc:729] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c467f9054a0d4f158e148283e5c53ceb, State: Initialized, Role: FOLLOWER
I20260929 00:53:50.053721 23647 consensus_queue.cc:260] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:50.053831 23647 raft_consensus.cc:388] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:53:50.053920 23647 raft_consensus.cc:482] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:53:50.054015 23647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:53:50.055933 23647 raft_consensus.cc:504] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:50.056274 23647 leader_election.cc:302] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c467f9054a0d4f158e148283e5c53ceb; no voters:
I20260929 00:53:50.056612 23647 leader_election.cc:288] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:53:50.056703 23651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:53:50.056977 23651 raft_consensus.cc:686] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [term 2 LEADER]: Becoming Leader. State: Replica: c467f9054a0d4f158e148283e5c53ceb, State: Running, Role: LEADER
I20260929 00:53:50.057350 23651 consensus_queue.cc:237] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } }
I20260929 00:53:50.057948 23647 sys_catalog.cc:564] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:50.058867 23652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c467f9054a0d4f158e148283e5c53ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } } }
I20260929 00:53:50.059008 23652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:50.059535 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c467f9054a0d4f158e148283e5c53ceb. Latest consensus state: current_term: 2 leader_uuid: "c467f9054a0d4f158e148283e5c53ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467f9054a0d4f158e148283e5c53ceb" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 35239 } } }
I20260929 00:53:50.059696 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:50.059841 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:53:50.060243 23647 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:53:50.066895 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:53:50.068617 23660 catalog_manager.cc:1259] Loaded cluster ID: 7ef8c70776144cc9b86ea17ac2341527
I20260929 00:53:50.068761 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:53:50.072346 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:53:50.073264 23660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Loaded TSK: 0
I20260929 00:53:50.074957 23660 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:53:50.183669 23649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:50.184119 23649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:50.184224 23649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:50.190718 23649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:50.190821 23649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:50.190917 23649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:53:50.199726 23649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:37327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=43023
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:50.201496 23649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:50.203020 23649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:50.217232 23685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.217234 23687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.217417 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:50.218593 23649 server_base.cc:1034] running on GCE node
I20260929 00:53:50.219138 23649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:50.220069 23649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:50.221271 23649 hybrid_clock.cc:648] HybridClock initialized: now 1790643230221247 us; error 40 us; skew 500 ppm
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:50.225450 23649 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:53:50.227047 23649 webserver.cc:460] Webserver started at http://127.11.40.193:43023/ using document root <none> and password file <none>
I20260929 00:53:50.227979 23649 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:50.228168 23649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:50.278780 23649 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.000s
I20260929 00:53:50.289054 23695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.290971 23649 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260929 00:53:50.291122 23649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "9d114bf0c0634fc6b6b652652d3e1acb"
format_stamp: "Formatted at 2026-09-29 00:53:48 on dist-test-slave-5309"
server_key: "42a05b91d98fe9d724c173ff4876a344"
server_key_iv: "55a034656ae59943ca6faddf950bdded"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:50.291898 23649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:50.312783 23649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:50.317363 23649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:50.317617 23649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:50.318497 23649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:50.319690 23649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:50.319767 23649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.319842 23649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:50.319867 23649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.333273 23649 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:37327
I20260929 00:53:50.334065 23808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:37327 every 8 connection(s)
I20260929 00:53:50.334944 23649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:53:50.343729 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23649
I20260929 00:53:50.344275 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:39689
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:35239
--webserver_port=44135
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:50.360523 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.338195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.193:45175)
0929 00:53:50.339457 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:50.339524 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:50.340604 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:50.342994 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:50.343004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:50.343404 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:50.344161 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.344188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.346395 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.346402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.348029 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.348044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.349629 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.349636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.349911 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.350770 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:50.350801 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:50.352970 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:50.356232 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.356240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.356263 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.356713 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.357274 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.357280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.357286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.357529 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.358448 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:50.358455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:50.358805 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 00:53:50.359027 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 00:53:50.359414 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":171,"threads_started":1}
I20260929 00:53:50.361495 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.338772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:45175 (local address 127.11.40.254:35239)
0929 00:53:50.341100 (+ 2328us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:50.341124 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:50.341174 (+ 50us) server_negotiation.cc:407] Connection header received
0929 00:53:50.341489 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:50.341506 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:50.341958 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:50.342316 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:50.344437 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.346183 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.348204 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.349120 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.349331 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.353319 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:50.353332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:50.353354 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:50.353397 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:50.355921 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.356929 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.356938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.356944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.357121 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.357675 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.357679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.357682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.358237 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:50.358357 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:50.359384 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 00:53:50.359870 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":94,"threads_started":1}
I20260929 00:53:50.364660 23809 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:50.364970 23809 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.365646 23809 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
W20260929 00:53:50.368187 23809 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:35239 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 00:53:50.368350 23809 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.368561 23809 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:50.369475 23809 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.369648 23809 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
W20260929 00:53:50.370532 23809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:53:50.522311 23813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:50.522805 23813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:50.522930 23813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:50.529592 23813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:50.529757 23813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:50.529881 23813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:53:50.538321 23813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:39689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=44135
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:50.540040 23813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:50.541785 23813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:50.555750 23819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.555751 23818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.555872 23821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:50.556124 23813 server_base.cc:1034] running on GCE node
I20260929 00:53:50.556674 23813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:50.557446 23813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:50.573024 23813 hybrid_clock.cc:648] HybridClock initialized: now 1790643230569723 us; error 3286 us; skew 500 ppm
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:50.578012 23813 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:53:50.579403 23813 webserver.cc:460] Webserver started at http://127.11.40.194:44135/ using document root <none> and password file <none>
I20260929 00:53:50.580236 23813 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:50.580351 23813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:50.610620 23813 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 00:53:50.614850 23830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.616727 23813 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:53:50.616911 23813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "78932ab64c254dbda140735f343eb297"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "1eea4e562d9951ff3a3482879fa9fa9c"
server_key_iv: "7b9a87288f9378329a67f0a918684711"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:50.617730 23813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:50.653116 23813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:50.657110 23813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:50.657356 23813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:50.658087 23813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:50.659175 23813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:50.659250 23813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.659307 23813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:50.659338 23813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.670778 23813 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:39689
I20260929 00:53:50.670866 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:39689 every 8 connection(s)
I20260929 00:53:50.671965 23813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:53:50.675853 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23813
I20260929 00:53:50.676352 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.40.212:50397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:44073
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:35239
--webserver_port=44077
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:50.688732 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.674229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:52525 (local address 127.11.40.254:35239)
0929 00:53:50.674406 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:50.674412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:50.675338 (+ 926us) server_negotiation.cc:407] Connection header received
0929 00:53:50.676421 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:50.676425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:50.676487 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:50.676567 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:50.677777 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.678317 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.680113 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.680453 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.680521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.683687 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:50.683692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:50.683695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:50.683725 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:50.685623 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.686567 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.686571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.686573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.686749 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.687256 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.687259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.687261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.687549 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:50.687623 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:50.688414 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 00:53:50.688558 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:53:50.690611 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.674702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.194:52525)
0929 00:53:50.675171 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:50.675226 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:50.676209 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:50.676763 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:50.676775 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:50.677114 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:50.677623 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.677637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.678451 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.678458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.679967 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.679980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.680696 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.680699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.680960 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.681669 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:50.681690 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:50.683461 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:50.686087 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.686095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.686113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.686413 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.686878 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.686881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.686883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.687125 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.687714 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:50.687721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:50.688080 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 00:53:50.689184 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0929 00:53:50.689551 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260929 00:53:50.692071 23944 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:50.692379 23944 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.693063 23944 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
W20260929 00:53:50.695070 23944 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:35239 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 00:53:50.695222 23944 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.695454 23944 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:50.696484 23944 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.696666 23944 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
W20260929 00:53:50.697615 23944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:53:50.826488 23947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:50.826949 23947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:50.827052 23947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:50.833459 23947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:50.833559 23947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:50.833662 23947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:53:50.841903 23947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.40.212:50397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:44073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=44077
--tserver_master_addrs=127.11.40.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:50.843621 23947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:50.845160 23947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:50.859866 23953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.859910 23955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:50.860838 23952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:50.861339 23947 server_base.cc:1034] running on GCE node
I20260929 00:53:50.862004 23947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:50.862921 23947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:50.864208 23947 hybrid_clock.cc:648] HybridClock initialized: now 1790643230864187 us; error 63 us; skew 500 ppm
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:50.868026 23947 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:53:50.869346 23947 webserver.cc:460] Webserver started at http://127.11.40.195:44077/ using document root <none> and password file <none>
I20260929 00:53:50.870164 23947 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:50.870350 23947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:50.891352 23947 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:53:50.894611 23970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.896477 23947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 00:53:50.896627 23947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "71ed85404e6f4ea6866de0e15b9370e2"
format_stamp: "Formatted at 2026-09-29 00:53:49 on dist-test-slave-5309"
server_key: "770d0c4cec39caf5f4f7a5d81283f2f2"
server_key_iv: "8e6b7d68a7baa1e51c30cb3b82ca064c"
server_key_version: "kuduclusterkey@0"
I20260929 00:53:50.897426 23947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:50.912740 23947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:50.916707 23947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:50.916965 23947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:50.917806 23947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:50.918906 23947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:50.918969 23947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.919028 23947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:50.919066 23947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:50.930356 23947 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:44073
I20260929 00:53:50.930501 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:44073 every 8 connection(s)
I20260929 00:53:50.931457 23947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:53:50.936028 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 23947
Sep 29 00:53:50 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:50.948400 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.933600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:48653 (local address 127.11.40.254:35239)
0929 00:53:50.933756 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:50.933761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:50.934701 (+ 940us) server_negotiation.cc:407] Connection header received
0929 00:53:50.935691 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:50.935695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:50.935762 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:50.935871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:50.937807 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.938706 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.940082 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.940426 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.940544 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.943473 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:50.943477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:50.943480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:50.943511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:50.945823 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.946563 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.946566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.946568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.946652 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.947059 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.947062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.947064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.947258 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:50.947369 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:50.948059 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 00:53:50.948196 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":896}
I20260929 00:53:50.949432 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.937186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.11.40.254:35239)
0929 00:53:50.938432 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:50.938450 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:50.938467 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:53:50.938507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:50.938510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:50.938554 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:50.938654 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:50.939665 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.940352 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.941517 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.941910 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.942021 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.944017 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:50.944022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:50.944025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:50.944080 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:50.946246 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.947403 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.947406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.947407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.947481 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:50.947802 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:50.947805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:50.947807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:50.948034 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:50.948230 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:50.949005 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 00:53:50.949128 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1047,"spinlock_wait_cycles":7936,"thread_start_us":114,"threads_started":1}
I20260929 00:53:50.949542 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:50.934046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:35239 (local address 127.11.40.195:48653)
0929 00:53:50.934508 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:50.934563 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:50.935477 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:50.936084 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:50.936095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:50.936469 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:50.937540 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.937570 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.938864 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.938867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.939935 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:50.939947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:50.940610 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:50.940614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:50.940854 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:50.941594 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:50.941615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:50.943294 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:50.945980 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.945988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.946010 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.946427 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.946746 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:50.946750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:50.946752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:50.946966 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:50.947369 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:50.947378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:50.947732 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 00:53:50.947980 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 00:53:50.948364 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260929 00:53:50.951012 24084 heartbeater.cc:347] Connected to a master server at 127.11.40.254:35239
I20260929 00:53:50.951282 24084 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:50.951925 24084 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:50.953589 23611 ts_manager.cc:194] Registered new tserver with Master: 71ed85404e6f4ea6866de0e15b9370e2 (127.11.40.195:44073)
I20260929 00:53:50.954810 23611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:48653
Sep 29 00:53:51 dist-test-slave-5309 krb5kdc[22614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643231, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:51.370819 23809 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:51.371152 23809 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:51.371976 23610 ts_manager.cc:194] Registered new tserver with Master: 9d114bf0c0634fc6b6b652652d3e1acb (127.11.40.193:37327)
I20260929 00:53:51.372798 23610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:45175
Sep 29 00:53:51 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643231, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
I20260929 00:53:51.697842 23944 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:51.698180 23944 heartbeater.cc:511] Master 127.11.40.254:35239 requested a full tablet report, sending...
I20260929 00:53:51.699180 23610 ts_manager.cc:194] Registered new tserver with Master: 78932ab64c254dbda140735f343eb297 (127.11.40.194:39689)
I20260929 00:53:51.699980 23610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:52525
I20260929 00:53:51.704473 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:53:51.717070 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.706302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.11.40.254:35239)
0929 00:53:51.706912 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.706917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.707277 (+ 360us) server_negotiation.cc:407] Connection header received
0929 00:53:51.707322 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.707325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.707373 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.707457 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:51.708455 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.709318 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.710552 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.710909 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.711014 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.712876 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:51.712880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:51.712883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:51.712911 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:51.714811 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:51.715358 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:51.715361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:51.715364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:51.715466 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:51.715817 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:51.715821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:51.715823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:51.716064 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:51.716222 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.716517 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.716667 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":100,"threads_started":1}
I20260929 00:53:51.721961 23610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:53:51.727170 23610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:53:51.748102 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.741104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:44073 (local address 127.0.0.1:40052)
0929 00:53:51.741824 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:51.741839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:51.741935 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:51.742959 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:51.742966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:51.742994 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:53:51.743146 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.743155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.744277 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.744281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.745658 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.745666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.747510 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.747514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.747670 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.747720 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 00:53:51.747816 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 00:53:51.747939 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":84,"threads_started":1}
I20260929 00:53:51.748944 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.741653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.11.40.195:44073)
0929 00:53:51.742434 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.742439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.742458 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:53:51.742523 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.742527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.742783 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.743324 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:53:51.743453 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.744157 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.745802 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.747371 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.748437 (+ 1066us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.748617 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.748676 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.748799 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":97,"threads_started":1}
I20260929 00:53:51.752004 24018 tablet_service.cc:1467] Processing CreateTablet for tablet e70a1407bf994b09be6258e24e3908a4 (DEFAULT_TABLE table=test-table [id=5aab7e11d3894a728842a2488c07012b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:53:51.753396 24018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70a1407bf994b09be6258e24e3908a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:51.753567 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.741375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:37327 (local address 127.0.0.1:33716)
0929 00:53:51.744410 (+ 3035us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:51.744424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:51.744513 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:51.745227 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:51.745230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:51.745245 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:53:51.745346 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.745351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.746624 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.746628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.747716 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.747724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.753118 (+ 5394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.753123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.753230 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.753261 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:53:51.753331 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 00:53:51.753410 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2973,"thread_start_us":86,"threads_started":1}
I20260929 00:53:51.755075 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.742015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.11.40.193:37327)
0929 00:53:51.742303 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.742310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.744608 (+ 2298us) server_negotiation.cc:407] Connection header received
0929 00:53:51.744691 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.744696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.744962 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.745099 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:53:51.745469 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.746493 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.747851 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.752706 (+ 4855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.753843 (+ 1137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.754007 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.754062 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.754160 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
I20260929 00:53:51.758141 23730 tablet_service.cc:1467] Processing CreateTablet for tablet e70a1407bf994b09be6258e24e3908a4 (DEFAULT_TABLE table=test-table [id=5aab7e11d3894a728842a2488c07012b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:53:51.759349 23730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70a1407bf994b09be6258e24e3908a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:51.759559 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.739430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39689 (local address 127.0.0.1:41326)
0929 00:53:51.739795 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:51.739841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:51.739960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:51.753564 (+ 13604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:51.753567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:51.753590 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:53:51.753712 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.753720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.756668 (+ 2948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.756671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.757832 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.757843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.759123 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.759127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.759225 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.759252 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:53:51.759305 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 00:53:51.759422 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":114,"threads_started":1}
I20260929 00:53:51.761193 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.740244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.11.40.194:39689)
0929 00:53:51.753035 (+ 12791us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.753043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.753071 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:53:51.753161 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.753166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.753373 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.753525 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:53:51.755361 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.756524 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.758032 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.758945 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.760695 (+ 1750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.760891 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.760950 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.761055 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12540,"thread_start_us":83,"threads_started":1}
I20260929 00:53:51.764094 23878 tablet_service.cc:1467] Processing CreateTablet for tablet e70a1407bf994b09be6258e24e3908a4 (DEFAULT_TABLE table=test-table [id=5aab7e11d3894a728842a2488c07012b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:53:51.765228 23878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70a1407bf994b09be6258e24e3908a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:51.773478 24110 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Bootstrap starting.
I20260929 00:53:51.775888 24110 tablet_bootstrap.cc:654] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:51.776778 24110 log.cc:824] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:51.778928 24110 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: No bootstrap required, opened a new log
I20260929 00:53:51.779114 24110 ts_tablet_manager.cc:1397] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 00:53:51.781337 24111 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Bootstrap starting.
I20260929 00:53:51.783758 24111 tablet_bootstrap.cc:654] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:51.783794 24110 raft_consensus.cc:348] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.784221 24110 raft_consensus.cc:374] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:51.784317 24110 raft_consensus.cc:729] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ed85404e6f4ea6866de0e15b9370e2, State: Initialized, Role: FOLLOWER
I20260929 00:53:51.784677 24111 log.cc:824] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:51.784947 24110 consensus_queue.cc:260] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.785909 24113 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Bootstrap starting.
I20260929 00:53:51.786098 24110 ts_tablet_manager.cc:1428] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260929 00:53:51.786512 24084 heartbeater.cc:503] Master 127.11.40.254:35239 was elected leader, sending a full tablet report...
I20260929 00:53:51.788203 24113 tablet_bootstrap.cc:654] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:51.788964 24111 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: No bootstrap required, opened a new log
I20260929 00:53:51.789209 24111 ts_tablet_manager.cc:1397] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 00:53:51.789238 24113 log.cc:824] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:51.791975 24113 tablet_bootstrap.cc:492] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: No bootstrap required, opened a new log
I20260929 00:53:51.792191 24113 ts_tablet_manager.cc:1397] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:53:51.793275 24111 raft_consensus.cc:348] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.793535 24111 raft_consensus.cc:374] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:51.793578 24111 raft_consensus.cc:729] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78932ab64c254dbda140735f343eb297, State: Initialized, Role: FOLLOWER
I20260929 00:53:51.794070 24111 consensus_queue.cc:260] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.795171 24111 ts_tablet_manager.cc:1428] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:53:51.795656 23944 heartbeater.cc:503] Master 127.11.40.254:35239 was elected leader, sending a full tablet report...
I20260929 00:53:51.797400 24113 raft_consensus.cc:348] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.797773 24113 raft_consensus.cc:374] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:51.797866 24113 raft_consensus.cc:729] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d114bf0c0634fc6b6b652652d3e1acb, State: Initialized, Role: FOLLOWER
I20260929 00:53:51.798409 24113 consensus_queue.cc:260] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.799512 24113 ts_tablet_manager.cc:1428] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:53:51.799767 23809 heartbeater.cc:503] Master 127.11.40.254:35239 was elected leader, sending a full tablet report...
W20260929 00:53:51.837690 23810 tablet.cc:2369] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:53:51.924324 23945 tablet.cc:2369] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 00:53:51 dist-test-slave-5309 krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643231, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for HTTP/127.11.40.212@KRBTEST.COM
W20260929 00:53:51.934098 24085 tablet.cc:2369] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:53:51.952937 24118 raft_consensus.cc:482] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:53:51.953210 24118 raft_consensus.cc:504] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.954761 24118 leader_election.cc:288] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78932ab64c254dbda140735f343eb297 (127.11.40.194:39689), 71ed85404e6f4ea6866de0e15b9370e2 (127.11.40.195:44073)
I20260929 00:53:51.960958 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.955609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:56751 (local address 127.11.40.194:39689)
0929 00:53:51.955746 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.955751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.956061 (+ 310us) server_negotiation.cc:407] Connection header received
0929 00:53:51.956357 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.956361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.956420 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.956479 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:53:51.957037 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.957883 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.959522 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.960138 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.960243 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.960299 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.960698 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.960791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:53:51.961453 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.954969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39689 (local address 127.11.40.193:56751)
0929 00:53:51.955967 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:51.955984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:51.956155 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:51.956646 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:51.956649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:51.956683 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:53:51.956861 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.956871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.958040 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.958043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.959369 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.959381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.960407 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.960411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.960523 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.960548 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:53:51.961206 (+ 658us) client_negotiation.cc:240] Negotiation successful
0929 00:53:51.961295 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":126,"threads_started":1}
I20260929 00:53:51.962319 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70a1407bf994b09be6258e24e3908a4" candidate_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78932ab64c254dbda140735f343eb297" is_pre_election: true
I20260929 00:53:51.962700 23897 raft_consensus.cc:2454] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d114bf0c0634fc6b6b652652d3e1acb in term 0.
I20260929 00:53:51.963366 23698 leader_election.cc:302] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78932ab64c254dbda140735f343eb297, 9d114bf0c0634fc6b6b652652d3e1acb; no voters:
I20260929 00:53:51.963737 24118 raft_consensus.cc:2781] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:53:51.963838 24118 raft_consensus.cc:482] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:53:51.963888 24118 raft_consensus.cc:3037] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:51.965379 24118 raft_consensus.cc:504] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.965842 24118 leader_election.cc:288] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [CANDIDATE]: Term 1 election: Requested vote from peers 78932ab64c254dbda140735f343eb297 (127.11.40.194:39689), 71ed85404e6f4ea6866de0e15b9370e2 (127.11.40.195:44073)
I20260929 00:53:51.966464 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70a1407bf994b09be6258e24e3908a4" candidate_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78932ab64c254dbda140735f343eb297"
I20260929 00:53:51.966609 23897 raft_consensus.cc:3037] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:51.968024 23897 raft_consensus.cc:2454] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d114bf0c0634fc6b6b652652d3e1acb in term 1.
I20260929 00:53:51.969456 23698 leader_election.cc:302] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78932ab64c254dbda140735f343eb297, 9d114bf0c0634fc6b6b652652d3e1acb; no voters:
I20260929 00:53:51.969686 24118 raft_consensus.cc:2781] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:53:51.969974 24118 raft_consensus.cc:686] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [term 1 LEADER]: Becoming Leader. State: Replica: 9d114bf0c0634fc6b6b652652d3e1acb, State: Running, Role: LEADER
I20260929 00:53:51.970382 24118 consensus_queue.cc:237] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } }
I20260929 00:53:51.975548 23610 catalog_manager.cc:5495] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d114bf0c0634fc6b6b652652d3e1acb (127.11.40.193). New cstate: current_term: 1 leader_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 37327 } health_report { overall_health: HEALTHY } } }
I20260929 00:53:51.980297 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.955247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:59235 (local address 127.11.40.195:44073)
0929 00:53:51.955595 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:51.955600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:51.962549 (+ 6949us) server_negotiation.cc:407] Connection header received
0929 00:53:51.962846 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:51.962849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:51.962910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:51.962965 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:53:51.968238 (+ 5273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.969117 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.978796 (+ 9679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.979434 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.979542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.979598 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:51.980057 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 00:53:51.980156 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260929 00:53:51.980823 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:51.955134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:44073 (local address 127.11.40.193:59235)
0929 00:53:51.962452 (+ 7318us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:51.962469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:51.962697 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:51.965410 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:51.965413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:51.965439 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:53:51.965859 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.965869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.977398 (+ 11529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.977405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.978602 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:51.978618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:51.979721 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:51.979724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:51.979836 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:51.979860 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:53:51.980534 (+ 674us) client_negotiation.cc:240] Negotiation successful
0929 00:53:51.980629 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7239,"thread_start_us":126,"threads_started":1}
I20260929 00:53:51.981482 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70a1407bf994b09be6258e24e3908a4" candidate_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ed85404e6f4ea6866de0e15b9370e2" is_pre_election: true
I20260929 00:53:51.981824 24038 raft_consensus.cc:2454] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d114bf0c0634fc6b6b652652d3e1acb in term 0.
I20260929 00:53:51.982298 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70a1407bf994b09be6258e24e3908a4" candidate_uuid: "9d114bf0c0634fc6b6b652652d3e1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ed85404e6f4ea6866de0e15b9370e2"
I20260929 00:53:51.982461 24037 raft_consensus.cc:3037] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:51.983839 24037 raft_consensus.cc:2454] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d114bf0c0634fc6b6b652652d3e1acb in term 1.
I20260929 00:53:52.041790 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:52.035612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.11.40.193:37327)
0929 00:53:52.035771 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:52.035777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:52.035838 (+ 61us) server_negotiation.cc:407] Connection header received
0929 00:53:52.036043 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:52.036047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:52.036118 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:52.036252 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:53:52.036625 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:52.037575 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:52.039361 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:52.039705 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:52.039811 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:52.040286 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:53:52.040946 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:53:52.041067 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:52.041546 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 00:53:52.041615 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:53:52.055732 24037 raft_consensus.cc:1261] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 1 FOLLOWER]: Refusing update from remote peer 9d114bf0c0634fc6b6b652652d3e1acb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:53:52.055819 23897 raft_consensus.cc:1261] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 1 FOLLOWER]: Refusing update from remote peer 9d114bf0c0634fc6b6b652652d3e1acb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:53:52.056525 24118 consensus_queue.cc:1035] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ed85404e6f4ea6866de0e15b9370e2" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:53:52.056895 24122 consensus_queue.cc:1035] T e70a1407bf994b09be6258e24e3908a4 P 9d114bf0c0634fc6b6b652652d3e1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "78932ab64c254dbda140735f343eb297" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:53:52.071298 24127 mvcc.cc:205] Tried to move back new op lower bound from 7334474678489116672 to 7334474678154465280. Current Snapshot: MvccSnapshot[applied={T|T < 7334474678489116672}]
I20260929 00:53:52.086772 23610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:53:52.086982 23610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:53:52.090479 23610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c467f9054a0d4f158e148283e5c53ceb: Sending DeleteTablet for 3 replicas of tablet e70a1407bf994b09be6258e24e3908a4
I20260929 00:53:52.091854 23878 tablet_service.cc:1514] Processing DeleteTablet for tablet e70a1407bf994b09be6258e24e3908a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:53:52 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:41326
I20260929 00:53:52.092484 24018 tablet_service.cc:1514] Processing DeleteTablet for tablet e70a1407bf994b09be6258e24e3908a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:53:52 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:40052
I20260929 00:53:52.093041 23730 tablet_service.cc:1514] Processing DeleteTablet for tablet e70a1407bf994b09be6258e24e3908a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:53:52 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:33716
I20260929 00:53:52.093482 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23649
I20260929 00:53:52.101081 24135 tablet_replica.cc:330] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: stopping tablet replica
I20260929 00:53:52.101540 24135 raft_consensus.cc:2229] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:53:52.101999 24135 raft_consensus.cc:2258] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:53:52.102501 24134 tablet_replica.cc:330] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: stopping tablet replica
I20260929 00:53:52.102880 24134 raft_consensus.cc:2229] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:53:52.103180 24134 pending_rounds.cc:62] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Trying to abort 2 pending ops.
I20260929 00:53:52.103276 24134 pending_rounds.cc:68] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334474678489116672 op_type: WRITE_OP write_request { tablet_id: "e70a1407bf994b09be6258e24e3908a4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\314\232\354\325\006\"8\n\ntest-admin\022*\n 5aab7e11d3894a728842a2488c07012b\020\001\030\001 \001(\001" signature: "<redacted>""\033\213\237\232\307<\201\337\202\300\037\324\315\347\271\346\000\311w7\025\324/K_*\305[\027\243\232>^/\275\'\341\250\032\252\311nY\362K\305\203D\013\265k\244\354\344{(1h<\260EZ\372\366" signing_key_seq_num: 0 } } request_id { client_id: "676fd6de3b3c42ac948d0200a05c0da4" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260929 00:53:52.104313 24134 pending_rounds.cc:68] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334474678154465280 op_type: NO_OP noop_request { }
I20260929 00:53:52.104439 24134 raft_consensus.cc:2866] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 00:53:52.104537 24134 raft_consensus.cc:2258] T e70a1407bf994b09be6258e24e3908a4 P 78932ab64c254dbda140735f343eb297 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 00:53:52.111992 23833 connection.cc:535] server connection from 127.11.40.193:56751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 00:53:52.112685 23598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:37327 (error 108)
I20260929 00:53:52.114439 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:52.114008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:37327 (local address 127.0.0.1:33738)
0929 00:53:52.114245 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:52.114337 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:37327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 00:53:52.114426 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23813
W20260929 00:53:52.114684 23598 catalog_manager.cc:4584] TS 9d114bf0c0634fc6b6b652652d3e1acb (127.11.40.193:37327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e70a1407bf994b09be6258e24e3908a4: Network error: Client connection negotiation failed: client connection to 127.11.40.193:37327: connect: Connection refused (error 111)
I20260929 00:53:52.115020 24135 ts_tablet_manager.cc:1905] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:53:52.118733 24135 ts_tablet_manager.cc:1918] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:53:52.118853 24135 log.cc:1196] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/e70a1407bf994b09be6258e24e3908a4
I20260929 00:53:52.119277 24135 ts_tablet_manager.cc:1939] T e70a1407bf994b09be6258e24e3908a4 P 71ed85404e6f4ea6866de0e15b9370e2: Deleting consensus metadata
I20260929 00:53:52.120787 23596 catalog_manager.cc:4841] TS 71ed85404e6f4ea6866de0e15b9370e2 (127.11.40.195:44073): tablet e70a1407bf994b09be6258e24e3908a4 (table test-table [id=5aab7e11d3894a728842a2488c07012b]) successfully deleted
W20260929 00:53:52.122893 24104 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:53:52.122410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39689 (local address 127.0.0.1:41342)
0929 00:53:52.122523 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:52.122539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:52.122658 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:52.122760 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:39689: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 00:53:52.122973 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23947
I20260929 00:53:52.133260 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 23578
I20260929 00:53:52.141831 11427 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 00:53:52.487437 11427 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 00:53:52.487560 11427 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:53:52.833104 11427 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:53:52.833 UTC [22644] LOG: received smart shutdown request
2026-09-29 00:53:52.837 UTC [22644] LOG: background worker "logical replication launcher" (PID 22653) exited with exit code 1
2026-09-29 00:53:52.838 UTC [22648] LOG: shutting down
2026-09-29 00:53:53.038 UTC [22644] LOG: database system is shut down
2026-09-29T00:53:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52550 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:53:53 dist-test-slave-5309 krb5kdc[24146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:53:53 dist-test-slave-5309 krb5kdc[24146](info): set up 2 sockets
Sep 29 00:53:53 dist-test-slave-5309 krb5kdc[24146](info): commencing operation
krb5kdc: starting...
W20260929 00:53:54.250573 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:53:54.386400 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43379
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:54.505823 24159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:54.506116 24159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:54.506167 24159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:54.510294 24159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:54.510356 24159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:54.510376 24159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:54.510392 24159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:54.510408 24159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:54.515805 24159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:54.516994 24159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:54.518071 24159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:54.527211 24167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:54.527230 24164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:54.527281 24159 server_base.cc:1034] running on GCE node
W20260929 00:53:54.527235 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:54.528136 24159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:54.529630 24159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:54.531088 24159 hybrid_clock.cc:648] HybridClock initialized: now 1790643234531013 us; error 91 us; skew 500 ppm
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:54.534556 24159 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:53:54.535616 24159 webserver.cc:460] Webserver started at http://127.11.40.254:44513/ using document root <none> and password file <none>
I20260929 00:53:54.536213 24159 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:54.536276 24159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:54.536525 24159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:54.538781 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "7cc1dc4b682f4df183146aaa0db2d946"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "522a445b1ad93b0b68a2f4a9f5449933"
server_key_iv: "6068c15e27ade1d03013d83f8e787b7b"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.539403 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "7cc1dc4b682f4df183146aaa0db2d946"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "522a445b1ad93b0b68a2f4a9f5449933"
server_key_iv: "6068c15e27ade1d03013d83f8e787b7b"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.543234 24159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:53:54.545775 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:54.547000 24159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:53:54.547142 24159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "7cc1dc4b682f4df183146aaa0db2d946"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "522a445b1ad93b0b68a2f4a9f5449933"
server_key_iv: "6068c15e27ade1d03013d83f8e787b7b"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.547254 24159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:54.573745 24159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:54.577337 24159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:54.577571 24159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:54.585819 24159 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:43379
I20260929 00:53:54.585848 24226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:43379 every 8 connection(s)
I20260929 00:53:54.586907 24159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:53:54.590042 24227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:54.593791 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24159
I20260929 00:53:54.593937 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:53:54.594221 11427 external_mini_cluster.cc:1428] Setting key 78006e7130f311214288de83df6eb319
I20260929 00:53:54.594291 24227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.253@KRBTEST.COM" created.
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:54.613794 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.597485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.11.40.254:43379)
0929 00:53:54.597997 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:54.598017 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:54.598322 (+ 305us) server_negotiation.cc:407] Connection header received
0929 00:53:54.599503 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:54.599508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:54.599738 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:54.599945 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:54.600901 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.601789 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.602813 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.603262 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.603455 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.607491 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:54.607503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:54.607523 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:54.607574 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:54.610043 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.610723 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.610729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.610730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.610850 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.611324 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.611328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.611332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.611817 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:54.611936 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:54.612520 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 00:53:54.612797 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":342,"thread_start_us":160,"threads_started":1}
I20260929 00:53:54.613798 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.597485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43379 (local address 127.0.0.1:38774)
0929 00:53:54.598146 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.598187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:54.599136 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:54.599942 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:54.599947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:54.600438 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:54.600703 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.600720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.601925 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.601928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.602689 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.602698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.603441 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.603444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.603556 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.604301 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:54.604319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:54.607272 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:54.610189 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.610204 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.610220 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.610580 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.610964 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.610970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.610975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.611163 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.611965 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:54.611981 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:54.612210 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 00:53:54.612443 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 00:53:54.612912 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":47,"spinlock_wait_cycles":1792,"thread_start_us":182,"threads_started":1}
I20260929 00:53:54.616361 24227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:53:54.617235 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.616965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53028)
0929 00:53:54.617105 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.617173 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 00:53:54.617522 24176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
I20260929 00:53:54.618458 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.618253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53044)
0929 00:53:54.618363 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.618409 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 00:53:54.618610 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46283: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
I20260929 00:53:54.626528 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:46283
--webserver_interface=127.11.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:53:54.667886 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } attempt: 1
I20260929 00:53:54.668908 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.668535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53052)
0929 00:53:54.668703 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.668775 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:53:54.669862 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.669654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53064)
0929 00:53:54.669760 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.669805 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:53:54.669994 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46283: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
I20260929 00:53:54.738189 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } attempt: 2
I20260929 00:53:54.739149 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.738813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53080)
0929 00:53:54.738996 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.739061 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 00:53:54.739940 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.739715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53086)
0929 00:53:54.739826 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.739881 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:53:54.740108 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46283: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46283: connect: Connection refused (error 111)
W20260929 00:53:54.740839 24234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:54.741158 24234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:54.741252 24234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:54.745402 24234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:54.745479 24234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:54.745541 24234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:54.745601 24234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:54.745639 24234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:54.751060 24234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:54.752209 24234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:54.753335 24234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:54.762635 24240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:54.762635 24239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:54.762647 24242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:54.762804 24234 server_base.cc:1034] running on GCE node
I20260929 00:53:54.763449 24234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:54.764592 24234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:54.765772 24234 hybrid_clock.cc:648] HybridClock initialized: now 1790643234765761 us; error 31 us; skew 500 ppm
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:54.769022 24234 init.cc:377] Logged in from keytab as kudu/127.11.40.253@KRBTEST.COM (short username kudu)
I20260929 00:53:54.770062 24234 webserver.cc:460] Webserver started at http://127.11.40.253:43861/ using document root <none> and password file <none>
I20260929 00:53:54.770668 24234 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:54.770732 24234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:54.770967 24234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:54.773249 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data/instance:
uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "99df5bdc93e9288e96d5854d82c4fe8d"
server_key_iv: "c7a6045511ef6b15d44728a3a04efbb8"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.773849 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal/instance:
uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "99df5bdc93e9288e96d5854d82c4fe8d"
server_key_iv: "c7a6045511ef6b15d44728a3a04efbb8"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.777967 24234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:53:54.780571 24249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:54.781977 24234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:53:54.782112 24234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "99df5bdc93e9288e96d5854d82c4fe8d"
server_key_iv: "c7a6045511ef6b15d44728a3a04efbb8"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.782223 24234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:54.799544 24234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:54.803056 24234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:54.803274 24234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:54.811059 24234 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.253:46283
I20260929 00:53:54.811090 24301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.253:46283 every 8 connection(s)
I20260929 00:53:54.812036 24234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
I20260929 00:53:54.814733 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24234
I20260929 00:53:54.814868 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-1/wal/instance
I20260929 00:53:54.814934 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:54.815179 11427 external_mini_cluster.cc:1428] Setting key b3f571f6b9c302a4bcffaf67a8eed4a7
I20260929 00:53:54.819260 24302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.40.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.252@KRBTEST.COM" created.
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:54.831280 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } attempt: 3
I20260929 00:53:54.835021 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.821304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.11.40.254:43379)
0929 00:53:54.821453 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:54.821457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:54.822510 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 00:53:54.823679 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:54.823683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:54.823749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:54.823864 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:54.825291 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.825906 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.827280 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.827549 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.827673 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.830560 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:54.830569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:54.830571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:54.830596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:54.832478 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.833182 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.833186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.833188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.833316 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.833746 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.833750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.833752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.834002 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:54.834116 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:54.834714 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 00:53:54.834862 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:53:54.835981 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.821805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43379 (local address 127.0.0.1:38788)
0929 00:53:54.822340 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.822385 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:54.823429 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:54.824069 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:54.824081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:54.824458 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:54.825088 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.825116 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.826044 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.826051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.827131 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.827141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.827787 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.827792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.827961 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.828681 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:54.828699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:54.830379 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:54.832688 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.832693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.832705 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.833067 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.833427 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.833430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.833432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.833606 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.834127 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:54.834133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:54.834431 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 00:53:54.834615 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 00:53:54.834952 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":149,"threads_started":1}
I20260929 00:53:54.838001 24302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:53:54.843560 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.832232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.11.40.253:46283)
0929 00:53:54.832687 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:54.832828 (+ 141us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:54.832861 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:53:54.832908 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:54.832913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:54.833175 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:54.833346 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:54.833776 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.834754 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.835659 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.836234 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.836343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.839253 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:54.839264 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:54.839277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:54.839301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:54.841292 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.841750 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.841756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.841758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.841848 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.842179 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.842183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.842184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.842415 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:54.842560 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:54.842860 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 00:53:54.842997 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260929 00:53:54.843887 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:42083
--webserver_interface=127.11.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:53:54.844967 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.832094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53098)
0929 00:53:54.832326 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.832342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:54.832420 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:54.833326 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:54.833330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:54.833513 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:54.833639 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.833650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.834876 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.834879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.835542 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.835552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.836382 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.836389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.836589 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.837191 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:54.837202 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:54.839138 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:54.841389 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.841392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.841395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.841645 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.841943 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.841946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.841947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.842053 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.842532 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:54.842536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:54.842643 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 00:53:54.842753 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 00:53:54.842947 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 00:53:54.847297 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.838503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53104)
0929 00:53:54.838613 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.838628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:54.838706 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:54.839160 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:54.839162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:54.839428 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:54.839562 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.839574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.840372 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.840378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.841099 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:54.841108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.841663 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.841665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:54.841857 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.842324 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:54.842337 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:54.843719 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:54.845849 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.845855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.845858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.846098 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.846375 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:54.846377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:54.846379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:54.846454 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:54.846796 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:54.846799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:54.846874 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 00:53:54.847009 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 00:53:54.847136 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":52}
I20260929 00:53:54.847335 24227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:53:54.847870 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.838605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.11.40.253:46283)
0929 00:53:54.838939 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:54.838947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:54.838958 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:53:54.838989 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:54.838992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:54.839036 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:54.839131 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:54.839699 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.840269 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.841216 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:54.841560 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:54.841701 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:54.843867 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:54.843871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:54.843873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:54.843901 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:54.845742 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.846199 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.846201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.846203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.846286 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:54.846537 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:54.846539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:54.846541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:54.846711 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:54.846800 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:54.847083 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:53:54.847210 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":271,"spinlock_wait_cycles":768,"thread_start_us":134,"threads_started":1}
I20260929 00:53:54.848060 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.847817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59398)
0929 00:53:54.847936 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.847990 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 00:53:54.848688 24302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:53:54.849138 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.848537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59408)
0929 00:53:54.848984 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.849048 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260929 00:53:54.849392 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.849143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59420)
0929 00:53:54.849251 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.849324 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:53:54.849393 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
W20260929 00:53:54.849576 24252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:54.850443 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.850202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59422)
0929 00:53:54.850309 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.850379 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:53:54.850564 24302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:54.880564 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 1
I20260929 00:53:54.881549 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.881200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59424)
0929 00:53:54.881383 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.881457 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 00:53:54.882371 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.882131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59440)
0929 00:53:54.882251 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.882303 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 00:53:54.882561 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:54.899829 24302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 1
I20260929 00:53:54.900878 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.900474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59452)
0929 00:53:54.900633 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.900733 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 00:53:54.901605 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.901397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59454)
0929 00:53:54.901503 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.901556 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 00:53:54.901742 24302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:54.957779 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 2
I20260929 00:53:54.958712 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.958397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59470)
0929 00:53:54.958557 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.958632 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 00:53:54.959538 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.959299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59482)
0929 00:53:54.959423 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.959482 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 00:53:54.959715 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
W20260929 00:53:54.962726 24309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:54.963063 24309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:54.963176 24309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:54.967444 24309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:54.967535 24309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:54.967572 24309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:54.967630 24309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:54.967672 24309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:54.969914 24302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 2
I20260929 00:53:54.970818 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.970513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59490)
0929 00:53:54.970660 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.970731 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:53:54.971567 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:54.971354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59506)
0929 00:53:54.971464 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:54.971515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 00:53:54.971697 24302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:54.973824 24309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:43379,127.11.40.253:46283,127.11.40.252:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:54.974992 24309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:54.976049 24309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:54.984853 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:54.984846 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:54.985212 24309 server_base.cc:1034] running on GCE node
W20260929 00:53:54.984931 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:54.986184 24309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:54.987401 24309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:54.988615 24309 hybrid_clock.cc:648] HybridClock initialized: now 1790643234988589 us; error 37 us; skew 500 ppm
Sep 29 00:53:54 dist-test-slave-5309 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:54.991940 24309 init.cc:377] Logged in from keytab as kudu/127.11.40.252@KRBTEST.COM (short username kudu)
I20260929 00:53:54.993001 24309 webserver.cc:460] Webserver started at http://127.11.40.252:40079/ using document root <none> and password file <none>
I20260929 00:53:54.993579 24309 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:54.993649 24309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:54.993834 24309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:54.995993 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data/instance:
uuid: "db7e15c32e894c3ab8231335090fa443"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "ce1f609fe341f187dbbe56175d75e0e4"
server_key_iv: "033f3091549114a5c234bca9d03339ea"
server_key_version: "encryptionkey@0"
I20260929 00:53:54.996583 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal/instance:
uuid: "db7e15c32e894c3ab8231335090fa443"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "ce1f609fe341f187dbbe56175d75e0e4"
server_key_iv: "033f3091549114a5c234bca9d03339ea"
server_key_version: "encryptionkey@0"
I20260929 00:53:55.001044 24309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:53:55.003720 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:55.005353 24309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:53:55.005519 24309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "db7e15c32e894c3ab8231335090fa443"
format_stamp: "Formatted at 2026-09-29 00:53:54 on dist-test-slave-5309"
server_key: "ce1f609fe341f187dbbe56175d75e0e4"
server_key_iv: "033f3091549114a5c234bca9d03339ea"
server_key_version: "encryptionkey@0"
I20260929 00:53:55.005687 24309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:55.056697 24309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:55.058888 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 3
I20260929 00:53:55.059757 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.059421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59508)
0929 00:53:55.059596 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.059661 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 00:53:55.060474 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.060243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59520)
0929 00:53:55.060357 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.060408 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 00:53:55.060564 24309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20260929 00:53:55.060657 24227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:55.060783 24309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:55.062868 24302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 3
I20260929 00:53:55.063588 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.063292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59524)
0929 00:53:55.063438 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.063505 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:53:55.064476 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.064119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59534)
0929 00:53:55.064334 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.064395 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260929 00:53:55.064647 24302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:42083: Network error: Client connection negotiation failed: client connection to 127.11.40.252:42083: connect: Connection refused (error 111)
I20260929 00:53:55.068337 24309 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.252:42083
I20260929 00:53:55.068343 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.252:42083 every 8 connection(s)
I20260929 00:53:55.069387 24309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
I20260929 00:53:55.072184 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:55.072559 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24309
I20260929 00:53:55.072685 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790642980693194-11427-0/minicluster-data/master-2/wal/instance
I20260929 00:53:55.073051 11427 external_mini_cluster.cc:1428] Setting key e4354ab5c96bdbadf1947c3d775fcace
I20260929 00:53:55.077517 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:55.089165 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.074524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.11.40.252:42083)
0929 00:53:55.075000 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.075015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.075052 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:53:55.075782 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.075802 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.076144 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.076432 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.077432 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.078567 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.079846 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.080514 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.080679 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.083383 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.083389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.083399 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.083430 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.085528 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.085985 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.085991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.085998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.086152 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.086571 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.086576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.086578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.087100 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.087258 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.087681 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.087988 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":164,"threads_started":1}
I20260929 00:53:55.089337 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.079536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:43379 (local address 127.0.0.1:38804)
0929 00:53:55.079961 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.080010 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:55.080153 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:55.080443 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:55.080448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:55.081009 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:55.081166 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.081172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.081981 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.081985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.082748 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.082757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.083256 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.083260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.083369 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.083975 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:55.083988 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:55.085620 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:55.087581 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.087588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.087594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.087934 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.088236 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.088239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.088241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.088345 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.088722 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:55.088728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:55.088860 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 00:53:55.089009 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:53:55.089178 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":896,"thread_start_us":179,"threads_started":1}
I20260929 00:53:55.089331 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.079618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.11.40.254:43379)
0929 00:53:55.079785 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.079790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.080113 (+ 323us) server_negotiation.cc:407] Connection header received
0929 00:53:55.080250 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.080253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.080303 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.080394 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.081330 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.081854 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.082877 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.083103 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.083217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.085850 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.085853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.085854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.085880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.087479 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.088047 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.088049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.088051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.088127 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.088437 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.088439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.088440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.088620 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.088699 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.089048 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.089163 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:53:55.090713 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:55.099864 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.091506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.11.40.253:46283)
0929 00:53:55.091683 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.091690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.091734 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:53:55.091852 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.091855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.091905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.091992 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.092383 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.092974 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.093723 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.093964 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.094077 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.096278 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.096281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.096284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.096315 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.098203 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.098646 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.098650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.098652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.098741 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.099009 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.099014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.099016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.099265 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.099353 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.099602 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.099729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 00:53:55.099861 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.091427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46283 (local address 127.0.0.1:53112)
0929 00:53:55.091675 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.091690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:55.091773 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:55.092005 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:55.092007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:55.092176 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:55.092264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.092269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.093081 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.093087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.093596 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.093604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.094075 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.094079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.094174 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.094658 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:55.094677 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:55.096137 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:55.098330 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.098333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.098334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.098535 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.098824 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.098827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.098829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.098913 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.099345 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:55.099348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:55.099442 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 00:53:55.099551 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:53:55.099688 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 00:53:55.100637 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.091545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.11.40.254:43379)
0929 00:53:55.091662 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.091666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.091726 (+ 60us) server_negotiation.cc:407] Connection header received
0929 00:53:55.091864 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.091867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.091913 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.091996 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.092597 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.093151 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.093924 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.094141 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.094258 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.096919 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.096924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.096927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.096952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.098655 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.099082 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.099085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.099087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.099174 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.099516 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.099519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.099521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.099721 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.099963 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.100302 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.100437 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 00:53:55.100858 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:53:55.110740 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.101374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59562)
0929 00:53:55.101485 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.101498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:55.101567 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:55.101834 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:55.101840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:55.102029 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:55.102118 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.102125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.103697 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.103701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.104475 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.104483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.105029 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.105033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.105103 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.105563 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:55.105573 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:55.106973 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:55.108913 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.108916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.108919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.109214 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.109527 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.109529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.109567 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.109684 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.110131 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:55.110134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:55.110240 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 00:53:55.110364 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 00:53:55.110502 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 00:53:55.110774 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.101416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.11.40.252:42083)
0929 00:53:55.101515 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.101520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.101533 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:53:55.101651 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.101654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.101687 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.101786 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.102262 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.103579 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.104584 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.104924 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.105034 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.107101 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.107105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.107107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.107137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.108785 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.109325 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.109331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.109336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.109429 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.109797 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.109801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.109803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.110036 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.110131 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.110417 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.110599 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":38}
I20260929 00:53:55.111800 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.101980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.11.40.253:46283)
0929 00:53:55.102097 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.102102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.102117 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:53:55.102217 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.102222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.102265 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.102335 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.103102 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.103845 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.104709 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.105075 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.105186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.108041 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.108047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.108053 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.108111 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.109986 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.110473 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.110477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.110478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.110582 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.110894 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.110898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.110899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.111103 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.111293 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.111483 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.111617 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:53:55.114465 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: Bootstrap starting.
I20260929 00:53:55.115396 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:53:55.117408 24378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:55.118368 24378 log.cc:824] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:55.120767 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: No bootstrap required, opened a new log
I20260929 00:53:55.125258 24378 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.125743 24378 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:55.125823 24378 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7e15c32e894c3ab8231335090fa443, State: Initialized, Role: FOLLOWER
I20260929 00:53:55.126442 24378 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.128208 24378 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:55.128616 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.117909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.11.40.254:43379)
0929 00:53:55.118033 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.118037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.118527 (+ 490us) server_negotiation.cc:407] Connection header received
0929 00:53:55.118779 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.118783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.118838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.118906 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.119983 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.120746 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.122845 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.123172 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.123285 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.124990 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.124994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.124996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.125025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.126665 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.127174 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.127178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.127180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.127269 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.127569 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.127573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.127593 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.127816 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.127921 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.128310 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.128443 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:53:55.129393 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.118614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.11.40.252:42083)
0929 00:53:55.118750 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.118754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.119106 (+ 352us) server_negotiation.cc:407] Connection header received
0929 00:53:55.119291 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.119296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.119347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.119401 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.120275 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.120773 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.122261 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.122610 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.122723 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.124774 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.124799 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.124801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.124858 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.127102 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.127628 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.127632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.127634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.127725 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.128011 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.128014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.128016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.128314 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.128414 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.129083 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.129204 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:53:55.131304 24391 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.131671 24391 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 00:53:55.134948 24402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:53:55.135080 24402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:53:55.135121 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.118234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.11.40.253:46283)
0929 00:53:55.118349 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.118355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.119109 (+ 754us) server_negotiation.cc:407] Connection header received
0929 00:53:55.119155 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.119158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.119203 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.119284 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.125687 (+ 6403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.126622 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.127815 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.128147 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.128264 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.130593 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.130597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.130599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.130633 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.132990 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.133465 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.133468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.133470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.133551 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.133848 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.133852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.133853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.134072 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.134184 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.134823 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.134944 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:53:55.132627 24309 master_runner.cc:186] Error getting master registration for 127.11.40.254:43379: OK, instance_id { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" instance_seqno: 1790643234583457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:53:55.136049 24309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:53:55.207864 24302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 4
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:53:55.218312 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.208582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59592)
0929 00:53:55.209168 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.209186 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:55.209339 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:55.209879 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:55.209882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:55.210101 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:55.210200 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.210205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.211030 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.211034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.211641 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.211650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.212193 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.212198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.212293 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.212936 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:55.212957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:55.214592 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:55.216669 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.216672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.216674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.216970 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.217290 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.217293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.217295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.217380 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.217866 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:55.217868 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:55.217964 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 00:53:55.218049 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:53:55.218171 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477}
I20260929 00:53:55.218551 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.209492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.11.40.252:42083)
0929 00:53:55.209638 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.209642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.209656 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:53:55.209698 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.209700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.209740 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.209857 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.210358 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.210900 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.211792 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.212028 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.212176 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.214773 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.214778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.214781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.214811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.216546 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.217067 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.217071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.217073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.217165 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.217503 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.217506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.217508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.217746 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.217850 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.218137 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.218261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:53:55.221534 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: Bootstrap starting.
I20260929 00:53:55.223994 24302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:55.224840 24302 log.cc:824] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:55.224854 24227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } attempt: 4
I20260929 00:53:55.228363 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: No bootstrap required, opened a new log
Sep 29 00:53:55 dist-test-slave-5309 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643234, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:53:55.232933 24302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.233198 24302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:55.233259 24302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed4b7dd1fc7486a89004a8b94d9d25a, State: Initialized, Role: FOLLOWER
I20260929 00:53:55.233769 24302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.234431 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.225382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:42083 (local address 127.0.0.1:59594)
0929 00:53:55.225553 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:55.225575 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:55.225687 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:55.226045 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:55.226049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:55.226250 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:55.226364 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.226372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.227391 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.227395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.228151 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:55.228163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.228696 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.228699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:55.228788 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.229420 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:55.229435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:55.230848 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:55.232687 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.232690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.232693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.232984 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.233331 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:55.233334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:55.233337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:55.233433 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:55.233876 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:55.233879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:55.233986 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 00:53:55.234095 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:53:55.234256 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 00:53:55.234431 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:55.225671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.11.40.252:42083)
0929 00:53:55.225786 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:55.225791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:55.225805 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:53:55.225851 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:55.225854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:55.225901 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:55.225997 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:55.226543 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.227254 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.228260 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:55.228555 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:55.228679 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:55.230995 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:55.231000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:55.231004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:55.231029 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:55.232554 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.233086 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.233090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.233092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.233214 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:55.233574 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:55.233576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:55.233578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:55.233775 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:55.233884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:55.234166 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 00:53:55.234283 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:53:55.235033 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.235203 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.236078 24302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:55.237635 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: Bootstrap starting.
I20260929 00:53:55.240610 24227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:55.241549 24227 log.cc:824] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: Log is configured to *not* fsync() on all Append() calls
W20260929 00:53:55.243040 24416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:53:55.243122 24416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:53:55.244035 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: No bootstrap required, opened a new log
I20260929 00:53:55.240998 24234 master_runner.cc:186] Error getting master registration for 127.11.40.254:43379: OK, instance_id { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" instance_seqno: 1790643234583457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:53:55.244500 24234 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:53:55.248778 24227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.249153 24227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:55.249323 24227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc1dc4b682f4df183146aaa0db2d946, State: Initialized, Role: FOLLOWER
I20260929 00:53:55.249928 24227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.250636 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.250943 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.251818 24227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:53:55.259335 24430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:53:55.259560 24430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:53:55.263957 24159 master_runner.cc:263] No leader master found from master 7cc1dc4b682f4df183146aaa0db2d946
I20260929 00:53:55.264060 24159 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 00:53:55.305140 24255 tablet.cc:2369] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:53:55.312232 24331 tablet.cc:2369] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:53:55.329665 24180 tablet.cc:2369] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:53:55.464864 24405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:53:55.465153 24405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.466192 24405 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc1dc4b682f4df183146aaa0db2d946 (127.11.40.254:43379), db7e15c32e894c3ab8231335090fa443 (127.11.40.252:42083)
I20260929 00:53:55.466810 24201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc1dc4b682f4df183146aaa0db2d946" is_pre_election: true
I20260929 00:53:55.466835 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7e15c32e894c3ab8231335090fa443" is_pre_election: true
I20260929 00:53:55.467177 24352 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed4b7dd1fc7486a89004a8b94d9d25a in term 0.
I20260929 00:53:55.467183 24201 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed4b7dd1fc7486a89004a8b94d9d25a in term 0.
I20260929 00:53:55.467720 24252 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ed4b7dd1fc7486a89004a8b94d9d25a, 7cc1dc4b682f4df183146aaa0db2d946, db7e15c32e894c3ab8231335090fa443; no voters:
I20260929 00:53:55.468031 24405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:53:55.468164 24405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:53:55.468212 24405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:55.469712 24405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.470100 24405 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [CANDIDATE]: Term 1 election: Requested vote from peers 7cc1dc4b682f4df183146aaa0db2d946 (127.11.40.254:43379), db7e15c32e894c3ab8231335090fa443 (127.11.40.252:42083)
I20260929 00:53:55.470420 24201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc1dc4b682f4df183146aaa0db2d946"
I20260929 00:53:55.470559 24201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:55.470772 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7e15c32e894c3ab8231335090fa443"
I20260929 00:53:55.470916 24352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:55.471655 24201 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed4b7dd1fc7486a89004a8b94d9d25a in term 1.
I20260929 00:53:55.472035 24251 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed4b7dd1fc7486a89004a8b94d9d25a, 7cc1dc4b682f4df183146aaa0db2d946; no voters:
I20260929 00:53:55.472100 24352 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed4b7dd1fc7486a89004a8b94d9d25a in term 1.
I20260929 00:53:55.472262 24405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:53:55.472498 24405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [term 1 LEADER]: Becoming Leader. State: Replica: 3ed4b7dd1fc7486a89004a8b94d9d25a, State: Running, Role: LEADER
I20260929 00:53:55.472919 24405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } }
I20260929 00:53:55.474641 24432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed4b7dd1fc7486a89004a8b94d9d25a. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.474738 24432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:55.475121 24434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:53:55.476007 24434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:53:55.480899 24201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [term 1 FOLLOWER]: Refusing update from remote peer 3ed4b7dd1fc7486a89004a8b94d9d25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:53:55.480899 24352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [term 1 FOLLOWER]: Refusing update from remote peer 3ed4b7dd1fc7486a89004a8b94d9d25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:53:55.481675 24405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:53:55.481868 24432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:53:55.486344 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed4b7dd1fc7486a89004a8b94d9d25a. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.486344 24391 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed4b7dd1fc7486a89004a8b94d9d25a. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.486498 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.486498 24391 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.489305 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.489441 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:55.489543 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.489600 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:55.489866 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.489992 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.490738 24391 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc1dc4b682f4df183146aaa0db2d946" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 43379 } } peers { permanent_uuid: "3ed4b7dd1fc7486a89004a8b94d9d25a" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46283 } } peers { permanent_uuid: "db7e15c32e894c3ab8231335090fa443" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 42083 } } }
I20260929 00:53:55.490859 24391 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:53:55.491042 24434 catalog_manager.cc:1347] Generated new cluster ID: 618185260be84bfea734f04421c5d992
I20260929 00:53:55.491104 24434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:53:55.508876 24434 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:53:55.509951 24434 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:53:55.520143 24434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ed4b7dd1fc7486a89004a8b94d9d25a: Generated new TSK 0
I20260929 00:53:55.521343 24434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:53:56.136698 24402 catalog_manager.cc:1259] Loaded cluster ID: 618185260be84bfea734f04421c5d992
I20260929 00:53:56.136859 24402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: loading cluster ID for follower catalog manager: success
I20260929 00:53:56.138999 24402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: acquiring CA information for follower catalog manager: success
I20260929 00:53:56.140328 24402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db7e15c32e894c3ab8231335090fa443: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:53:56.261085 24430 catalog_manager.cc:1259] Loaded cluster ID: 618185260be84bfea734f04421c5d992
I20260929 00:53:56.261250 24430 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: loading cluster ID for follower catalog manager: success
I20260929 00:53:56.263216 24430 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: acquiring CA information for follower catalog manager: success
I20260929 00:53:56.264467 24430 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7cc1dc4b682f4df183146aaa0db2d946: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:53:56.579756 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24159
I20260929 00:53:56.586303 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24234
I20260929 00:53:56.592543 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24309
2026-09-29T00:53:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3416 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:53:56 dist-test-slave-5309 krb5kdc[24449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:53:56 dist-test-slave-5309 krb5kdc[24449](info): set up 2 sockets
Sep 29 00:53:56 dist-test-slave-5309 krb5kdc[24449](info): commencing operation
krb5kdc: starting...
W20260929 00:53:57.658499 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:53:57 dist-test-slave-5309 krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:53:57.798229 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:40719
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:40719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:57.914165 24462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:57.914443 24462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:57.914484 24462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:57.918567 24462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:57.918643 24462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:57.918663 24462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:57.918687 24462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:57.918702 24462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:57.924250 24462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:40719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:57.925480 24462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:57.926578 24462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:57.936067 24467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:57.936280 24468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:57.937044 24470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:57.937235 24462 server_base.cc:1034] running on GCE node
I20260929 00:53:57.937896 24462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:57.939159 24462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:57.940364 24462 hybrid_clock.cc:648] HybridClock initialized: now 1790643237940325 us; error 52 us; skew 500 ppm
Sep 29 00:53:57 dist-test-slave-5309 krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643237, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:57.943399 24462 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:53:57.944377 24462 webserver.cc:460] Webserver started at http://127.11.40.254:42325/ using document root <none> and password file <none>
I20260929 00:53:57.945078 24462 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:57.945175 24462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:57.945442 24462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:57.947590 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "2832f6ea4927496dad89f9939839a03d"
format_stamp: "Formatted at 2026-09-29 00:53:57 on dist-test-slave-5309"
server_key: "c6d1b37ef9e600f4fef2b8bf004c016b"
server_key_iv: "e298bc311559df4499cb0b750106a048"
server_key_version: "encryptionkey@0"
I20260929 00:53:57.948182 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "2832f6ea4927496dad89f9939839a03d"
format_stamp: "Formatted at 2026-09-29 00:53:57 on dist-test-slave-5309"
server_key: "c6d1b37ef9e600f4fef2b8bf004c016b"
server_key_iv: "e298bc311559df4499cb0b750106a048"
server_key_version: "encryptionkey@0"
I20260929 00:53:57.952160 24462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:53:57.954936 24477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:57.956535 24462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:53:57.956723 24462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "2832f6ea4927496dad89f9939839a03d"
format_stamp: "Formatted at 2026-09-29 00:53:57 on dist-test-slave-5309"
server_key: "c6d1b37ef9e600f4fef2b8bf004c016b"
server_key_iv: "e298bc311559df4499cb0b750106a048"
server_key_version: "encryptionkey@0"
I20260929 00:53:57.956898 24462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:57.974609 24462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:57.978015 24462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:57.978261 24462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:57.986119 24462 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:40719
I20260929 00:53:57.986137 24529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:40719 every 8 connection(s)
I20260929 00:53:57.987140 24462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:53:57.990106 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:57.995842 24530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: Bootstrap starting.
I20260929 00:53:57.996009 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24462
I20260929 00:53:57.996145 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:53:57.996438 11427 external_mini_cluster.cc:1428] Setting key ecfb9954d3cc2aded4d892952a662b41
I20260929 00:53:57.998869 24530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:57.999723 24530 log.cc:824] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:58.002173 24530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: No bootstrap required, opened a new log
Sep 29 00:53:58 dist-test-slave-5309 krb5kdc[24449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:58.007498 24530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } }
I20260929 00:53:58.007789 24530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:58.007874 24530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2832f6ea4927496dad89f9939839a03d, State: Initialized, Role: FOLLOWER
I20260929 00:53:58.008392 24530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } }
I20260929 00:53:58.008565 24530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:53:58.008665 24530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:53:58.008772 24530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:58.010152 24530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } }
I20260929 00:53:58.010543 24530 leader_election.cc:302] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2832f6ea4927496dad89f9939839a03d; no voters:
I20260929 00:53:58.010900 24530 leader_election.cc:288] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:53:58.011044 24535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:53:58.011272 24535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [term 1 LEADER]: Becoming Leader. State: Replica: 2832f6ea4927496dad89f9939839a03d, State: Running, Role: LEADER
I20260929 00:53:58.011662 24535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } }
I20260929 00:53:58.012383 24530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:58.013816 24537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2832f6ea4927496dad89f9939839a03d. Latest consensus state: current_term: 1 leader_uuid: "2832f6ea4927496dad89f9939839a03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } } }
I20260929 00:53:58.013815 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2832f6ea4927496dad89f9939839a03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2832f6ea4927496dad89f9939839a03d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 40719 } } }
I20260929 00:53:58.013948 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:57.997909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.11.40.254:40719)
0929 00:53:57.998589 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:57.998608 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:57.998651 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:53:57.999521 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:57.999549 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:57.999989 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:58.000430 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:58.001405 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:58.002565 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:58.003931 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:58.004413 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:58.004543 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:58.007848 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:58.007855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:58.007874 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:58.007916 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:58.010231 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:58.010822 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:58.010831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:58.010840 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:58.010997 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:58.011372 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:58.011375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:58.011377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:58.011905 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:58.012110 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:58.012471 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 00:53:58.012828 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":305,"threads_started":1}
I20260929 00:53:58.013973 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:58.013973 24537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:58.017285 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 00:53:58.018469 24550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:53:58.018550 24550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:53:58.018622 24546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:53:58.019471 24546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:53:58.025164 24546 catalog_manager.cc:1347] Generated new cluster ID: 3d1629526e264482947438e87c097987
I20260929 00:53:58.025250 24546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:53:58.031016 24546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:53:58.031984 24546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:53:58.042965 24546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2832f6ea4927496dad89f9939839a03d: Generated new TSK 0
I20260929 00:53:58.043769 24546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:53:58.073561 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24462
2026-09-29T00:53:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1479 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:53:58 dist-test-slave-5309 krb5kdc[24557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:53:58 dist-test-slave-5309 krb5kdc[24557](info): set up 2 sockets
Sep 29 00:53:58 dist-test-slave-5309 krb5kdc[24557](info): commencing operation
krb5kdc: starting...
W20260929 00:53:59.138890 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:53:59.273151 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:59.396193 24570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:59.396745 24570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:59.396905 24570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:59.401909 24570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:53:59.402133 24570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:59.402186 24570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:59.402228 24570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:53:59.402262 24570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:53:59.409297 24570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:59.410676 24570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:59.411828 24570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:53:59.421878 24570 server_base.cc:1034] running on GCE node
W20260929 00:53:59.421849 24576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:59.421832 24575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:59.421813 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:59.422781 24570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:59.424072 24570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:59.425362 24570 hybrid_clock.cc:648] HybridClock initialized: now 1790643239425295 us; error 87 us; skew 500 ppm
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:59.433977 24570 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:53:59.435051 24570 webserver.cc:460] Webserver started at http://127.11.40.254:40801/ using document root <none> and password file <none>
I20260929 00:53:59.435612 24570 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:59.435674 24570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:59.435851 24570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:59.438030 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "6843071baac54c0db74f8f3211bf1739"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "c5809c3c2077781df0244336a776b7f4"
server_key_iv: "f5b4634abfaa351a7568aabfaa2c7c4d"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.438582 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "6843071baac54c0db74f8f3211bf1739"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "c5809c3c2077781df0244336a776b7f4"
server_key_iv: "f5b4634abfaa351a7568aabfaa2c7c4d"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.442603 24570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:53:59.445465 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:59.447024 24570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:53:59.447184 24570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "6843071baac54c0db74f8f3211bf1739"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "c5809c3c2077781df0244336a776b7f4"
server_key_iv: "f5b4634abfaa351a7568aabfaa2c7c4d"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.447312 24570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:59.481314 24570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:59.484771 24570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:59.485031 24570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:59.492909 24570 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:37519
I20260929 00:53:59.492959 24637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:37519 every 8 connection(s)
I20260929 00:53:59.493934 24570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:53:59.497097 24638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:53:59.501511 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24570
I20260929 00:53:59.501675 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:53:59.501976 11427 external_mini_cluster.cc:1428] Setting key efaab6160a5d5237da0e691c8d5c9dde
I20260929 00:53:59.504686 24638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap starting.
I20260929 00:53:59.507427 24638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Neither blocks nor log segments found. Creating new log.
I20260929 00:53:59.508230 24638 log.cc:824] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Log is configured to *not* fsync() on all Append() calls
I20260929 00:53:59.510740 24638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: No bootstrap required, opened a new log
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:53:59.514515 24638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:53:59.514854 24638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:53:59.514952 24638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Initialized, Role: FOLLOWER
I20260929 00:53:59.515424 24638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:53:59.515588 24638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:53:59.515686 24638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:53:59.515841 24638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:53:59.517132 24638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:53:59.517553 24638 leader_election.cc:302] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6843071baac54c0db74f8f3211bf1739; no voters:
I20260929 00:53:59.517961 24638 leader_election.cc:288] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:53:59.518169 24643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:53:59.518421 24643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 LEADER]: Becoming Leader. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Running, Role: LEADER
I20260929 00:53:59.518844 24643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:53:59.519233 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:59.503988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.11.40.254:37519)
0929 00:53:59.504565 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:59.504582 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:59.504622 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:53:59.505390 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:59.505410 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:59.505780 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:59.506088 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:59.507040 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.508195 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.509367 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.509877 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.510011 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:59.512881 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:59.512887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:59.512902 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:59.512931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:59.515623 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:59.516214 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:59.516220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:59.516224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:59.516355 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:59.516691 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:59.516694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:59.516695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:59.517221 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:59.517431 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:59.517746 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 00:53:59.518167 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":260,"threads_started":1}
I20260929 00:53:59.519320 24638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:53:59.520156 24643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6843071baac54c0db74f8f3211bf1739. Latest consensus state: current_term: 1 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:53:59.520347 24643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:59.520589 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:53:59.520699 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:53:59.520756 24652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:53:59.523788 24652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:53:59.530093 24652 catalog_manager.cc:1347] Generated new cluster ID: 7db09604ea6b4f2cb3208bb2f5fbc71a
I20260929 00:53:59.530174 24652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:53:59.546136 24652 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:53:59.547857 24652 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:53:59.552917 24652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Generated new TSK 0
I20260929 00:53:59.553743 24652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:53:59.595793 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:59.725365 24664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:59.725651 24664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:59.725697 24664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:59.729815 24664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:59.729882 24664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:59.729985 24664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:53:59.735389 24664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:59.736583 24664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:59.737707 24664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:59.748510 24669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:59.748675 24664 server_base.cc:1034] running on GCE node
W20260929 00:53:59.748510 24672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:59.748553 24670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:59.749363 24664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:59.750036 24664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:59.751261 24664 hybrid_clock.cc:648] HybridClock initialized: now 1790643239751226 us; error 58 us; skew 500 ppm
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:59.754734 24664 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:53:59.755661 24664 webserver.cc:460] Webserver started at http://127.11.40.193:33897/ using document root <none> and password file <none>
I20260929 00:53:59.756250 24664 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:59.756343 24664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:59.756659 24664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:59.758819 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "1c5507585ce34f38b3a781990e217faa"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "b9132f06087fbe89b8a492858b045269"
server_key_iv: "6467be26244697e6813a9255a3a02d82"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.759390 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "1c5507585ce34f38b3a781990e217faa"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "b9132f06087fbe89b8a492858b045269"
server_key_iv: "6467be26244697e6813a9255a3a02d82"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.763254 24664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:53:59.765940 24679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:59.767303 24664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:53:59.767452 24664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "1c5507585ce34f38b3a781990e217faa"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "b9132f06087fbe89b8a492858b045269"
server_key_iv: "6467be26244697e6813a9255a3a02d82"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.767572 24664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:53:59.781998 24664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:53:59.785564 24664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:53:59.785779 24664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:53:59.786407 24664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:53:59.787271 24664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:53:59.787343 24664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:59.787412 24664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:53:59.787463 24664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:59.799736 24664 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45579
I20260929 00:53:59.800326 24792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45579 every 8 connection(s)
I20260929 00:53:59.800962 24664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:53:59.804292 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24664
I20260929 00:53:59.804443 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:53:59.804826 11427 external_mini_cluster.cc:1428] Setting key 9339052c225594a3928eb8afa12e7843
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:53:59.816802 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:59.802797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:53485 (local address 127.11.40.254:37519)
0929 00:53:59.802937 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:53:59.802941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:53:59.803766 (+ 825us) server_negotiation.cc:407] Connection header received
0929 00:53:59.805528 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:53:59.805531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:53:59.805587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:53:59.805677 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:53:59.807083 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.807665 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.809087 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.809334 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.809452 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:59.812248 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:53:59.812251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:53:59.812253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:53:59.812281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:53:59.814057 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:59.814941 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:59.814945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:59.814947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:59.815048 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:53:59.815502 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:53:59.815506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:53:59.815510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:53:59.815759 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:53:59.815878 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:53:59.816480 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 00:53:59.816628 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:53:59.817896 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:53:59.803161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:53485)
0929 00:53:59.803611 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 00:53:59.803656 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:53:59.805316 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:53:59.805841 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:53:59.805848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:53:59.806216 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:53:59.806863 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:59.806887 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.807808 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.807814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:59.808960 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:53:59.808970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:53:59.809493 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:53:59.809497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:53:59.809675 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:53:59.810357 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:53:59.810375 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:53:59.812026 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:53:59.814418 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:59.814432 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:59.814457 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:59.814817 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:59.815160 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:53:59.815165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:53:59.815169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:53:59.815377 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:53:59.815878 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:53:59.815887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:53:59.816181 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 00:53:59.816437 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 00:53:59.816847 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:53:59.819407 24793 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:53:59.819716 24793 heartbeater.cc:464] Registering TS with master...
I20260929 00:53:59.820348 24793 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:53:59.822180 24601 ts_manager.cc:194] Registered new tserver with Master: 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579)
I20260929 00:53:59.823895 24601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:53485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:53:59.837072 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:53:59.951704 24798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:53:59.952032 24798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:53:59.952078 24798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:53:59.956094 24798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:53:59.956171 24798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:53:59.956233 24798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:53:59.961522 24798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:53:59.962580 24798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:53:59.963585 24798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:53:59.973887 24804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:59.973923 24803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:53:59.973887 24806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:53:59.974184 24798 server_base.cc:1034] running on GCE node
I20260929 00:53:59.974663 24798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:53:59.975387 24798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:53:59.976609 24798 hybrid_clock.cc:648] HybridClock initialized: now 1790643239976571 us; error 50 us; skew 500 ppm
Sep 29 00:53:59 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:53:59.979614 24798 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:53:59.980506 24798 webserver.cc:460] Webserver started at http://127.11.40.194:32939/ using document root <none> and password file <none>
I20260929 00:53:59.981125 24798 fs_manager.cc:359] Metadata directory not provided
I20260929 00:53:59.981199 24798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:53:59.981372 24798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:53:59.983847 24798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "375ea74b09c3cc198d8bff768f91f41e"
server_key_iv: "81fd94351418db11bb6bb36f5623ab93"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.984409 24798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "375ea74b09c3cc198d8bff768f91f41e"
server_key_iv: "81fd94351418db11bb6bb36f5623ab93"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.988473 24798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:53:59.991369 24813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:53:59.992949 24798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:53:59.993072 24798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "375ea74b09c3cc198d8bff768f91f41e"
server_key_iv: "81fd94351418db11bb6bb36f5623ab93"
server_key_version: "encryptionkey@0"
I20260929 00:53:59.993176 24798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:00.012640 24798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:00.016345 24798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:00.016526 24798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:00.017187 24798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:00.018116 24798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:00.018167 24798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.018235 24798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:00.018268 24798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.028505 24798 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:43967
I20260929 00:54:00.028643 24926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:43967 every 8 connection(s)
I20260929 00:54:00.029541 24798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:00.034293 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24798
I20260929 00:54:00.034428 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:00.034709 11427 external_mini_cluster.cc:1428] Setting key 1d748d6123e9e633a7a1d55ca5bbde34
Sep 29 00:54:00 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:54:00.046224 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.031653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:53511 (local address 127.11.40.254:37519)
0929 00:54:00.031828 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.031834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.032735 (+ 901us) server_negotiation.cc:407] Connection header received
0929 00:54:00.034221 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.034225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.034293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.034418 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:00.035927 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.036585 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.038476 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.038935 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.039091 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.041827 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:00.041830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:00.041832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:00.041860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:00.043617 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.044275 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.044278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.044280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.044366 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.044914 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.044917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.044918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.045126 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:00.045295 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.045937 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.046080 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:54:00.047322 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.032040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:53511)
0929 00:54:00.032547 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.032613 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.033965 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.034650 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.034660 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.035182 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:00.035731 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.035774 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.036758 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.036765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.038320 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.038330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.039145 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.039148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.039309 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.040014 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:00.040031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:00.041610 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:00.043750 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:00.043757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:00.043776 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:00.044148 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:00.044471 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:00.044474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:00.044476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:00.044694 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:00.045315 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:00.045322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:00.045663 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.045830 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.046102 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:54:00.048674 24927 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:54:00.048965 24927 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:00.049502 24927 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:54:00.050736 24601 ts_manager.cc:194] Registered new tserver with Master: 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
I20260929 00:54:00.051390 24601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:53511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:54:00.067727 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:00.180517 24932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:00.180874 24932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:00.180919 24932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:00.185089 24932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:00.185165 24932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:00.185227 24932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:54:00.190919 24932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:00.192129 24932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:00.193259 24932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:00.203670 24940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.203651 24937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.203634 24938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:00.203943 24932 server_base.cc:1034] running on GCE node
I20260929 00:54:00.204473 24932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:00.205289 24932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:00.206488 24932 hybrid_clock.cc:648] HybridClock initialized: now 1790643240206454 us; error 45 us; skew 500 ppm
Sep 29 00:54:00 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:00.210065 24932 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:54:00.211092 24932 webserver.cc:460] Webserver started at http://127.11.40.195:45049/ using document root <none> and password file <none>
I20260929 00:54:00.211723 24932 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:00.211823 24932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:00.212085 24932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:00.214383 24932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "09d62e5553a84e65aa16bbbd8d5169b9"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-5309"
server_key: "14ff40cef7adb3ff58372cb0d4ec22de"
server_key_iv: "aba3c34cb021dba97bb12b39b5df4661"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.214980 24932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "09d62e5553a84e65aa16bbbd8d5169b9"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-5309"
server_key: "14ff40cef7adb3ff58372cb0d4ec22de"
server_key_iv: "aba3c34cb021dba97bb12b39b5df4661"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.219036 24932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:54:00.222368 24947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.224115 24932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:00.224255 24932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "09d62e5553a84e65aa16bbbd8d5169b9"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-5309"
server_key: "14ff40cef7adb3ff58372cb0d4ec22de"
server_key_iv: "aba3c34cb021dba97bb12b39b5df4661"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.224418 24932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:00.245169 24932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:00.249208 24932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:00.249466 24932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:00.250157 24932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:00.251078 24932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:00.251155 24932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.251231 24932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:00.251276 24932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.261344 24932 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:36915
I20260929 00:54:00.261365 25060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:36915 every 8 connection(s)
I20260929 00:54:00.262315 24932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:00.264976 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 24932
I20260929 00:54:00.265143 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:00.265511 11427 external_mini_cluster.cc:1428] Setting key 3ed56ae4dd8799d5721d069afec608f4
Sep 29 00:54:00 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:00.278472 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.264450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:48281 (local address 127.11.40.254:37519)
0929 00:54:00.264677 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.264684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.265661 (+ 977us) server_negotiation.cc:407] Connection header received
0929 00:54:00.267502 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.267505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.267563 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.267649 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:00.269136 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.269683 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.270938 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.271191 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.271311 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.274440 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:00.274444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:00.274446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:00.274484 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:00.275987 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.276601 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.276604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.276606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.276701 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.277127 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.277131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.277133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.277340 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:00.277453 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.278012 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.278194 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:54:00.279731 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.264876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:48281)
0929 00:54:00.265455 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.265519 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.267288 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.267877 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.267888 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.268360 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:00.268985 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.269002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.269816 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.269819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.270808 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.270825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.271355 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.271359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.271553 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.272244 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:00.272261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:00.274228 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:00.276139 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:00.276146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:00.276164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:00.276465 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:00.276851 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:00.276854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:00.276856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:00.277011 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:00.277454 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:00.277465 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:00.277745 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.277951 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.278439 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260929 00:54:00.281033 25061 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:54:00.281308 25061 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:00.281831 25061 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:54:00.282850 24601 ts_manager.cc:194] Registered new tserver with Master: 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915)
I20260929 00:54:00.283443 24601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:48281
I20260929 00:54:00.292696 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:00.303377 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.294966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.11.40.254:37519)
0929 00:54:00.295111 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.295117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.295244 (+ 127us) server_negotiation.cc:407] Connection header received
0929 00:54:00.295418 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.295422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.295474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.295568 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:00.296556 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.297284 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.298217 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.298442 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.298553 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.300114 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:00.300123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:00.300126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:00.300153 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:00.301591 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.302042 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.302047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.302049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.302133 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.302413 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.302417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.302419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.302651 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:00.302838 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.303075 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.303200 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:54:00.306663 24601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:00.309784 24601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:00.325940 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.319460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:36915 (local address 127.0.0.1:42876)
0929 00:54:00.320539 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.320552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.320702 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.321379 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.321385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.321421 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:00.321619 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.321626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.323122 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.323126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.324303 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.324311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.325337 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.325341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.325461 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.325503 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.325633 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.325768 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":38,"spinlock_wait_cycles":3456,"thread_start_us":100,"threads_started":1}
I20260929 00:54:00.325950 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.319685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.11.40.195:36915)
0929 00:54:00.320213 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.320225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.320635 (+ 410us) server_negotiation.cc:407] Connection header received
0929 00:54:00.320874 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.320879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.321029 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.321133 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:00.321751 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.322968 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.324410 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.325184 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.325324 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.325488 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.325666 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.325767 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":148,"threads_started":1}
I20260929 00:54:00.327008 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.319302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.0.0.1:42462)
0929 00:54:00.320214 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.320271 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.320519 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.321379 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.321385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.321455 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:00.321618 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.321626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.322795 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.322801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.325267 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.325277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.326583 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.326588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.326689 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.326711 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.326784 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.326866 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":1005,"spinlock_wait_cycles":1280,"thread_start_us":152,"threads_started":1}
I20260929 00:54:00.327135 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.320214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.0.0.1:33106)
0929 00:54:00.321166 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.321179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.321316 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.321902 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.321905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.321925 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:00.322018 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.322024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.324207 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.324211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.325394 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.325402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.326639 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.326643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.326750 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.326777 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.326862 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.326940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":1,"thread_start_us":130,"threads_started":1}
I20260929 00:54:00.327497 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.320618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.11.40.193:45579)
0929 00:54:00.321045 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.321050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.321281 (+ 231us) server_negotiation.cc:407] Connection header received
0929 00:54:00.321468 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.321475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.321750 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.321925 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:00.322120 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.324071 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.325512 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.326504 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.326612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.326751 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.326912 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.327005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":86,"threads_started":1}
I20260929 00:54:00.327642 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.319406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.11.40.194:43967)
0929 00:54:00.319963 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.319974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.320404 (+ 430us) server_negotiation.cc:407] Connection header received
0929 00:54:00.320733 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.320742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.320994 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.321134 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:00.321794 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.322655 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.325400 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.326252 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.326359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.326513 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.327340 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.327483 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":153,"threads_started":1}
I20260929 00:54:00.329798 24995 tablet_service.cc:1467] Processing CreateTablet for tablet a7384d9a3a9844388801f725c4757599 (DEFAULT_TABLE table=test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:00.330262 24722 tablet_service.cc:1467] Processing CreateTablet for tablet a7384d9a3a9844388801f725c4757599 (DEFAULT_TABLE table=test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:00.330446 24861 tablet_service.cc:1467] Processing CreateTablet for tablet a7384d9a3a9844388801f725c4757599 (DEFAULT_TABLE table=test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:00.331065 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7384d9a3a9844388801f725c4757599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:00.331278 24722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7384d9a3a9844388801f725c4757599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:00.331466 24861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7384d9a3a9844388801f725c4757599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:00.337289 25078 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Bootstrap starting.
I20260929 00:54:00.338701 25079 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Bootstrap starting.
I20260929 00:54:00.339797 25080 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Bootstrap starting.
I20260929 00:54:00.340461 25078 tablet_bootstrap.cc:654] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:00.341305 25079 tablet_bootstrap.cc:654] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:00.341470 25078 log.cc:824] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:00.342096 25079 log.cc:824] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:00.342303 25080 tablet_bootstrap.cc:654] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:00.343165 25080 log.cc:824] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:00.344173 25078 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: No bootstrap required, opened a new log
I20260929 00:54:00.344389 25078 ts_tablet_manager.cc:1397] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:54:00.344731 25079 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: No bootstrap required, opened a new log
I20260929 00:54:00.344964 25079 ts_tablet_manager.cc:1397] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 00:54:00.345247 25080 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: No bootstrap required, opened a new log
I20260929 00:54:00.345431 25080 ts_tablet_manager.cc:1397] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:54:00.348026 25078 raft_consensus.cc:348] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.348342 25078 raft_consensus.cc:374] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:00.348398 25078 raft_consensus.cc:729] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5507585ce34f38b3a781990e217faa, State: Initialized, Role: FOLLOWER
I20260929 00:54:00.348918 25078 consensus_queue.cc:260] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.349130 25079 raft_consensus.cc:348] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.349413 25079 raft_consensus.cc:374] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:00.349476 25079 raft_consensus.cc:729] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d62e5553a84e65aa16bbbd8d5169b9, State: Initialized, Role: FOLLOWER
I20260929 00:54:00.349366 25080 raft_consensus.cc:348] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.349637 25080 raft_consensus.cc:374] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:00.349710 25080 raft_consensus.cc:729] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c35f64cf76f40c5ab12a4767e9c69ef, State: Initialized, Role: FOLLOWER
I20260929 00:54:00.349738 24793 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
I20260929 00:54:00.349921 25079 consensus_queue.cc:260] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.350138 25080 consensus_queue.cc:260] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.350674 25061 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
I20260929 00:54:00.351038 25079 ts_tablet_manager.cc:1428] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:54:00.351094 25078 ts_tablet_manager.cc:1428] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 00:54:00.352104 24927 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
I20260929 00:54:00.352173 25080 ts_tablet_manager.cc:1428] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 00:54:00.513468 25062 tablet.cc:2369] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:00.530794 24928 tablet.cc:2369] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:00.552407 24794 tablet.cc:2369] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:00.557044 25085 raft_consensus.cc:482] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:00.557303 25085 raft_consensus.cc:504] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.558561 25085 leader_election.cc:288] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967), 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579)
I20260929 00:54:00.564205 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.559250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52979 (local address 127.11.40.194:43967)
0929 00:54:00.559493 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.559501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.559551 (+ 50us) server_negotiation.cc:407] Connection header received
0929 00:54:00.559647 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.559651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.559723 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.559808 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:00.560121 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.560933 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.562727 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.563386 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.563503 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.563557 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.563931 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.564045 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 00:54:00.564185 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.559478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:43243 (local address 127.11.40.193:45579)
0929 00:54:00.559694 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.559698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.559714 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:00.559902 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.559905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.559961 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.560025 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:00.560300 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.561082 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.562738 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.563359 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.563503 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.563557 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.563931 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.564045 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 00:54:00.564222 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.558914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.11.40.195:52979)
0929 00:54:00.559071 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.559087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.559232 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.559841 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.559845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.559863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:00.559982 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.559987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.561057 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.561061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.562532 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.562544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.563555 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.563560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.563726 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.563787 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.563891 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.564022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":114}
I20260929 00:54:00.564230 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.559228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.11.40.195:43243)
0929 00:54:00.559620 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.559639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:00.559761 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:00.560063 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:00.560068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:00.560090 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:00.560200 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.560204 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.561194 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.561197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.562532 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:00.562544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.563587 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.563591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:00.563793 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.563813 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 00:54:00.563891 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 00:54:00.564017 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":155,"thread_start_us":139,"threads_started":1}
I20260929 00:54:00.565088 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7384d9a3a9844388801f725c4757599" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" is_pre_election: true
I20260929 00:54:00.565085 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7384d9a3a9844388801f725c4757599" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5507585ce34f38b3a781990e217faa" is_pre_election: true
I20260929 00:54:00.565454 24746 raft_consensus.cc:2454] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 0.
I20260929 00:54:00.565449 24881 raft_consensus.cc:2454] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 0.
I20260929 00:54:00.566002 24948 leader_election.cc:302] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef; no voters:
I20260929 00:54:00.566366 25085 raft_consensus.cc:2781] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:00.566475 25085 raft_consensus.cc:482] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:00.566525 25085 raft_consensus.cc:3037] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:00.567703 25085 raft_consensus.cc:504] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.568120 25085 leader_election.cc:288] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967), 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579)
I20260929 00:54:00.568619 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7384d9a3a9844388801f725c4757599" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
I20260929 00:54:00.568763 24881 raft_consensus.cc:3037] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:00.568799 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7384d9a3a9844388801f725c4757599" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5507585ce34f38b3a781990e217faa"
I20260929 00:54:00.569005 24746 raft_consensus.cc:3037] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:00.570041 24881 raft_consensus.cc:2454] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 1.
I20260929 00:54:00.570096 24746 raft_consensus.cc:2454] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 1.
I20260929 00:54:00.570458 24948 leader_election.cc:302] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef; no voters:
I20260929 00:54:00.570840 25085 raft_consensus.cc:2781] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:00.571180 25085 raft_consensus.cc:686] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 LEADER]: Becoming Leader. State: Replica: 09d62e5553a84e65aa16bbbd8d5169b9, State: Running, Role: LEADER
I20260929 00:54:00.571628 25085 consensus_queue.cc:237] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:00.575471 24599 catalog_manager.cc:5495] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } health_report { overall_health: HEALTHY } } }
I20260929 00:54:00.629068 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.624791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.11.40.195:36915)
0929 00:54:00.624994 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.624998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.625015 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:54:00.625119 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.625122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.625182 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.625276 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:00.625599 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.626103 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.627006 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.627241 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.627330 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.627859 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:00.628439 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:00.628547 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.628780 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.628906 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 00:54:00.635603 24746 raft_consensus.cc:1261] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Refusing update from remote peer 09d62e5553a84e65aa16bbbd8d5169b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:00.635907 24881 raft_consensus.cc:1261] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Refusing update from remote peer 09d62e5553a84e65aa16bbbd8d5169b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:00.636482 25085 consensus_queue.cc:1035] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:00.636866 25088 consensus_queue.cc:1035] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:00.644960 25091 mvcc.cc:205] Tried to move back new op lower bound from 7334474713634504704 to 7334474713384042496. Current Snapshot: MvccSnapshot[applied={T|T < 7334474713634504704}]
I20260929 00:54:00.646162 25092 mvcc.cc:205] Tried to move back new op lower bound from 7334474713634504704 to 7334474713384042496. Current Snapshot: MvccSnapshot[applied={T|T < 7334474713634504704}]
I20260929 00:54:00.647099 25093 mvcc.cc:205] Tried to move back new op lower bound from 7334474713634504704 to 7334474713384042496. Current Snapshot: MvccSnapshot[applied={T|T < 7334474713634504704}]
I20260929 00:54:00.656538 24599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:00.656750 24599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:00.659547 24599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Sending DeleteTablet for 3 replicas of tablet a7384d9a3a9844388801f725c4757599
I20260929 00:54:00.660897 24722 tablet_service.cc:1514] Processing DeleteTablet for tablet a7384d9a3a9844388801f725c4757599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:00 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:33106
I20260929 00:54:00.661022 24861 tablet_service.cc:1514] Processing DeleteTablet for tablet a7384d9a3a9844388801f725c4757599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:00 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42462
I20260929 00:54:00.661075 24995 tablet_service.cc:1514] Processing DeleteTablet for tablet a7384d9a3a9844388801f725c4757599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:00 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42876
I20260929 00:54:00.661787 25101 tablet_replica.cc:330] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: stopping tablet replica
I20260929 00:54:00.661912 25102 tablet_replica.cc:330] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: stopping tablet replica
I20260929 00:54:00.662209 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24664
I20260929 00:54:00.662282 25102 raft_consensus.cc:2229] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:00.663610 25102 raft_consensus.cc:2258] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:00.664466 25101 raft_consensus.cc:2229] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:00.665109 25101 raft_consensus.cc:2258] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:00.665481 25102 ts_tablet_manager.cc:1905] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:00.668874 25102 ts_tablet_manager.cc:1918] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:00.668991 25102 log.cc:1196] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/a7384d9a3a9844388801f725c4757599
I20260929 00:54:00.669337 25102 ts_tablet_manager.cc:1939] T a7384d9a3a9844388801f725c4757599 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting consensus metadata
I20260929 00:54:00.670799 24589 catalog_manager.cc:4841] TS 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): tablet a7384d9a3a9844388801f725c4757599 (table test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]) successfully deleted
I20260929 00:54:00.671039 25101 ts_tablet_manager.cc:1905] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 00:54:00.671504 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:45579 (error 108)
I20260929 00:54:00.671803 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24798
I20260929 00:54:00.678027 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.677639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.0.0.1:33114)
0929 00:54:00.677822 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.677915 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 00:54:00.678328 24586 catalog_manager.cc:4584] TS 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7384d9a3a9844388801f725c4757599: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45579: connect: Connection refused (error 111)
I20260929 00:54:00.678823 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24932
I20260929 00:54:00.680693 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.678965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.0.0.1:42470)
0929 00:54:00.680564 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:00.680617 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:43967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1489}
I20260929 00:54:00.684667 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 24570
I20260929 00:54:00.691545 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--webserver_interface=127.11.40.254
--webserver_port=40801
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:00.815153 25104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:00.815608 25104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:00.815686 25104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:00.820251 25104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:00.820362 25104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:00.820384 25104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:00.820403 25104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:00.820425 25104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:00.826402 25104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:00.827584 25104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:00.828691 25104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:00.845819 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.845839 25111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.845841 25110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:00.845907 25104 server_base.cc:1034] running on GCE node
I20260929 00:54:00.846856 25104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:00.848470 25104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:00.849757 25104 hybrid_clock.cc:648] HybridClock initialized: now 1790643240849703 us; error 78 us; skew 500 ppm
Sep 29 00:54:00 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:00.853618 25104 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:54:00.854766 25104 webserver.cc:460] Webserver started at http://127.11.40.254:40801/ using document root <none> and password file <none>
I20260929 00:54:00.855443 25104 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:00.855571 25104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:00.860960 25104 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:54:00.864507 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.866626 25104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:54:00.866779 25104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "6843071baac54c0db74f8f3211bf1739"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "c5809c3c2077781df0244336a776b7f4"
server_key_iv: "f5b4634abfaa351a7568aabfaa2c7c4d"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.867591 25104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:00.897567 25104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:00.902763 25104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:00.902967 25104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:00.911598 25104 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:37519
I20260929 00:54:00.911937 25172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:37519 every 8 connection(s)
I20260929 00:54:00.913203 25104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:00.918318 25173 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:54:00.920279 25173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap starting.
I20260929 00:54:00.921013 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25104
I20260929 00:54:00.921500 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:45579
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=33897
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:00.940376 25173 log.cc:824] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:00.950847 25173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:00.951275 25173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap complete.
I20260929 00:54:00.956071 25173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:00.956687 25173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Initialized, Role: FOLLOWER
I20260929 00:54:00.957338 25173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:00.957499 25173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:00.957599 25173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:00.957728 25173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:54:00.959892 25173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:00.960274 25173 leader_election.cc:302] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6843071baac54c0db74f8f3211bf1739; no voters:
I20260929 00:54:00.960666 25173 leader_election.cc:288] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:54:00.960752 25177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:54:00.960990 25177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 LEADER]: Becoming Leader. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Running, Role: LEADER
I20260929 00:54:00.961311 25177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:00.962005 25173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:00.962822 25178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:54:00.962951 25178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:00.962878 25179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6843071baac54c0db74f8f3211bf1739. Latest consensus state: current_term: 2 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:54:00.963021 25179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:00.964489 25185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:00.969112 25185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:00.969835 25185 catalog_manager.cc:1259] Loaded cluster ID: 7db09604ea6b4f2cb3208bb2f5fbc71a
I20260929 00:54:00.969928 25185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:00.971204 25185 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 00:54:00.974753 25185 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 00:54:00.979938 25185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:00.980890 25185 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Loaded TSK: 0
I20260929 00:54:00.981942 25185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:54:01.061856 25175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.062227 25175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.062402 25175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.067732 25175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.067880 25175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.067991 25175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:54:01.075193 25175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:45579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=33897
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:01.076403 25175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.077564 25175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:54:01.089854 25175 server_base.cc:1034] running on GCE node
W20260929 00:54:01.089797 25201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.089797 25200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.090032 25203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.090553 25175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.091475 25175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.092705 25175 hybrid_clock.cc:648] HybridClock initialized: now 1790643241092686 us; error 34 us; skew 500 ppm
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.096568 25175 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:54:01.097689 25175 webserver.cc:460] Webserver started at http://127.11.40.193:33897/ using document root <none> and password file <none>
I20260929 00:54:01.098327 25175 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.098410 25175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.103124 25175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:01.106367 25210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.107996 25175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:01.108237 25175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "1c5507585ce34f38b3a781990e217faa"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "b9132f06087fbe89b8a492858b045269"
server_key_iv: "6467be26244697e6813a9255a3a02d82"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.108958 25175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.131366 25175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.135715 25175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.135967 25175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.136677 25175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:01.138053 25217 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:54:01.143383 25175 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:54:01.143517 25175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 00:54:01.143630 25175 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:54:01.146070 25175 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:54:01.146183 25175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:01.146582 25217 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Bootstrap starting.
I20260929 00:54:01.157912 25175 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45579
I20260929 00:54:01.157995 25324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45579 every 8 connection(s)
I20260929 00:54:01.159096 25175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:01.160470 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25175
I20260929 00:54:01.160972 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:43967
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=32939
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:01.166436 25217 log.cc:824] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:01.177557 25217 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:01.178027 25217 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Bootstrap complete.
I20260929 00:54:01.178717 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.161706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:35819 (local address 127.11.40.254:37519)
0929 00:54:01.162399 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.162427 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.163127 (+ 700us) server_negotiation.cc:407] Connection header received
0929 00:54:01.165064 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.165076 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.165428 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.165787 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.167233 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.168327 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.169637 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.170118 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.170323 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.173210 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.173215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.173229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.173261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.175240 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.175832 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.175843 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.175848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.175976 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.176347 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.176352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.176354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.176736 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.176915 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.177477 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.177890 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":124,"threads_started":1}
I20260929 00:54:01.178731 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.161730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:35819)
0929 00:54:01.162867 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.162951 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.164503 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.165860 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.165869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.166328 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.166997 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.167022 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.168509 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.168512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.169502 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.169513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.170288 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.170292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.170478 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.171180 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.171197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.172972 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.175390 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.175397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.175411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.175702 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.176082 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.176085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.176088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.176245 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.176884 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.176890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.177163 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.177357 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.177726 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":172,"threads_started":1}
I20260929 00:54:01.179492 25217 ts_tablet_manager.cc:1397] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260929 00:54:01.181321 25325 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:54:01.181567 25325 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:01.182080 25325 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:54:01.182603 25217 raft_consensus.cc:348] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:01.183291 25217 raft_consensus.cc:729] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5507585ce34f38b3a781990e217faa, State: Initialized, Role: FOLLOWER
I20260929 00:54:01.183871 25217 consensus_queue.cc:260] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:01.185144 25217 ts_tablet_manager.cc:1428] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 00:54:01.185151 25137 ts_manager.cc:194] Registered new tserver with Master: 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579)
I20260929 00:54:01.189397 25137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:35819
I20260929 00:54:01.192342 25325 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:54:01.202708 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.188758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.0.0.1:33120)
0929 00:54:01.189851 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.189996 (+ 145us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.190172 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.190997 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.191005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.191493 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.191644 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.191652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.192905 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.192908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.193971 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.193984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.194765 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.194768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.194882 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.195497 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.195512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.197387 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.200453 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.200472 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.200483 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.200878 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.201291 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.201294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.201296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.201398 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.202005 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.202012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.202153 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.202303 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.202523 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":174,"threads_started":1}
I20260929 00:54:01.202736 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.189056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.11.40.193:45579)
0929 00:54:01.189681 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.189695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.190134 (+ 439us) server_negotiation.cc:407] Connection header received
0929 00:54:01.190574 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.190581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.190807 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.190961 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.191790 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.192766 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.194086 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.194634 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.194725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.197923 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.197936 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.197951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.198146 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.200284 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.201028 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.201034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.201036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.201180 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.201546 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.201550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.201551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.201881 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.202000 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.202420 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.202582 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":504,"thread_start_us":120,"threads_started":1}
I20260929 00:54:01.204089 25259 tablet_service.cc:1514] Processing DeleteTablet for tablet a7384d9a3a9844388801f725c4757599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:00 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:33120
I20260929 00:54:01.205122 25334 tablet_replica.cc:330] stopping tablet replica
I20260929 00:54:01.205447 25334 raft_consensus.cc:2229] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:01.205718 25334 raft_consensus.cc:2258] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:01.206905 25334 ts_tablet_manager.cc:1905] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:01.211304 25334 ts_tablet_manager.cc:1918] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:01.211411 25334 log.cc:1196] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/a7384d9a3a9844388801f725c4757599
I20260929 00:54:01.211706 25334 ts_tablet_manager.cc:1939] T a7384d9a3a9844388801f725c4757599 P 1c5507585ce34f38b3a781990e217faa: Deleting consensus metadata
I20260929 00:54:01.214941 25121 catalog_manager.cc:4841] TS 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579): tablet a7384d9a3a9844388801f725c4757599 (table test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]) successfully deleted
W20260929 00:54:01.299832 25329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.300149 25329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.300225 25329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.304442 25329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.304544 25329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.304636 25329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:54:01.310267 25329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:43967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=32939
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:01.311537 25329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.312711 25329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:01.323215 25342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.323226 25339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.323246 25340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.323513 25329 server_base.cc:1034] running on GCE node
I20260929 00:54:01.323989 25329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.324676 25329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.325861 25329 hybrid_clock.cc:648] HybridClock initialized: now 1790643241325827 us; error 50 us; skew 500 ppm
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.329164 25329 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:54:01.330181 25329 webserver.cc:460] Webserver started at http://127.11.40.194:32939/ using document root <none> and password file <none>
I20260929 00:54:01.330818 25329 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.330945 25329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.335464 25329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 00:54:01.339018 25349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.340612 25329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:01.340790 25329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "375ea74b09c3cc198d8bff768f91f41e"
server_key_iv: "81fd94351418db11bb6bb36f5623ab93"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.341503 25329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.352783 25329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.356153 25329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.356385 25329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.357031 25329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:01.358214 25356 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:54:01.362707 25329 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:54:01.362777 25329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 00:54:01.362828 25329 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:54:01.364765 25329 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:54:01.364845 25329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260929 00:54:01.365119 25356 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Bootstrap starting.
I20260929 00:54:01.372479 25356 log.cc:824] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:01.375243 25329 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:43967
I20260929 00:54:01.375322 25464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:43967 every 8 connection(s)
I20260929 00:54:01.376227 25329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:01.381572 25356 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:01.382000 25356 tablet_bootstrap.cc:492] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Bootstrap complete.
I20260929 00:54:01.382359 25356 ts_tablet_manager.cc:1397] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260929 00:54:01.382776 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25329
I20260929 00:54:01.383279 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:36915
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=45049
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:01.385910 25356 raft_consensus.cc:348] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:01.386478 25356 raft_consensus.cc:729] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c35f64cf76f40c5ab12a4767e9c69ef, State: Initialized, Role: FOLLOWER
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:01.387053 25356 consensus_queue.cc:260] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } }
I20260929 00:54:01.388177 25356 ts_tablet_manager.cc:1428] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:54:01.391141 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.378289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56011 (local address 127.11.40.254:37519)
0929 00:54:01.378426 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.378430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.379060 (+ 630us) server_negotiation.cc:407] Connection header received
0929 00:54:01.380141 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.380145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.380205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.380312 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.381588 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.382237 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.383710 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.383967 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.384105 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.387363 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.387367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.387369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.387397 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.389005 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.389539 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.389549 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.389552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.389636 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.390004 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.390007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.390009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.390205 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.390317 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.390823 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.390966 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:54:01.392000 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.378419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:56011)
0929 00:54:01.378907 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.378948 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.379900 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.380468 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.380475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.380933 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.381420 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.381434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.382392 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.382395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.383570 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.383585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.384158 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.384163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.384354 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.385473 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.385489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.387140 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.389134 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.389140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.389155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.389420 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.389740 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.389743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.389745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.389892 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.390316 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.390324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.390584 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.390783 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.391127 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":160,"threads_started":1}
I20260929 00:54:01.393437 25465 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:54:01.393714 25465 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:01.394411 25465 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:54:01.396387 25137 ts_manager.cc:194] Registered new tserver with Master: 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
I20260929 00:54:01.397758 25137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:56011
I20260929 00:54:01.399997 25465 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.194@KRBTEST.COM
I20260929 00:54:01.408146 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.397440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.0.0.1:42480)
0929 00:54:01.397652 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.397667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.397747 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.398516 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.398520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.398734 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.398836 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.398843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.400411 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.400418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.401235 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.401245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.402185 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.402188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.402260 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.402860 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.402878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.404190 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.406515 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.406518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.406519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.406730 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.407055 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.407058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.407059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.407170 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.407697 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.407699 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.407783 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.407869 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.407988 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 00:54:01.408370 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.397568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.11.40.194:43967)
0929 00:54:01.398030 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.398040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.398063 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:54:01.398135 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.398140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.398368 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.398507 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.398965 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.400255 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.401473 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.402059 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.402171 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.404351 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.404361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.404378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.404404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.406397 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.406849 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.406855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.406857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.406954 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.407262 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.407266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.407267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.407579 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.407721 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.407977 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.408171 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":129,"threads_started":1}
I20260929 00:54:01.409472 25398 tablet_service.cc:1514] Processing DeleteTablet for tablet a7384d9a3a9844388801f725c4757599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:00 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42480
I20260929 00:54:01.410499 25471 tablet_replica.cc:330] stopping tablet replica
I20260929 00:54:01.410815 25471 raft_consensus.cc:2229] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:01.411082 25471 raft_consensus.cc:2258] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:01.412088 25471 ts_tablet_manager.cc:1905] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:01.416210 25471 ts_tablet_manager.cc:1918] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:01.416301 25471 log.cc:1196] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/a7384d9a3a9844388801f725c4757599
I20260929 00:54:01.416579 25471 ts_tablet_manager.cc:1939] T a7384d9a3a9844388801f725c4757599 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting consensus metadata
I20260929 00:54:01.418341 25121 catalog_manager.cc:4841] TS 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): tablet a7384d9a3a9844388801f725c4757599 (table test-table [id=e4797c4c08324bb0a129e4f0c69f6bd8]) successfully deleted
W20260929 00:54:01.506883 25468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.507227 25468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.507323 25468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.511610 25468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.511720 25468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.511813 25468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:54:01.517441 25468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=45049
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:01.518548 25468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.519589 25468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:54:01.530051 25468 server_base.cc:1034] running on GCE node
W20260929 00:54:01.529973 25476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.530097 25477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.529973 25479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.530592 25468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.531350 25468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.532624 25468 hybrid_clock.cc:648] HybridClock initialized: now 1790643241532598 us; error 50 us; skew 500 ppm
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.535784 25468 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:54:01.536749 25468 webserver.cc:460] Webserver started at http://127.11.40.195:45049/ using document root <none> and password file <none>
I20260929 00:54:01.537364 25468 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.537478 25468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.541523 25468 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:01.543782 25486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.545254 25468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:01.545387 25468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "09d62e5553a84e65aa16bbbd8d5169b9"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-5309"
server_key: "14ff40cef7adb3ff58372cb0d4ec22de"
server_key_iv: "aba3c34cb021dba97bb12b39b5df4661"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.545950 25468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.565523 25468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.569159 25468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.569382 25468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.570062 25468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:01.570931 25468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:01.571004 25468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.571076 25468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:01.571130 25468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.581768 25468 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:36915
I20260929 00:54:01.581794 25599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:36915 every 8 connection(s)
I20260929 00:54:01.582945 25468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:01.593271 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25468
I20260929 00:54:01.598055 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.584923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:38201 (local address 127.11.40.254:37519)
0929 00:54:01.585079 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.585083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.585972 (+ 889us) server_negotiation.cc:407] Connection header received
0929 00:54:01.587367 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.587370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.587428 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.587511 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.588848 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.589398 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.590635 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.590875 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.590997 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.593941 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.593947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.593950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.593986 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.595706 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.596247 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.596250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.596252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.596343 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.596869 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.596872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.596874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.597067 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.597151 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.597780 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.597910 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:54:01.599220 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.585323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:38201)
0929 00:54:01.585811 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.585853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.587144 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.587748 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.587755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.588139 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.588662 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.588676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.589564 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.589567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.590488 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.590498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.591054 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.591059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.591239 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.591902 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.591919 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.593608 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.595840 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.595846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.595861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.596134 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.596468 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.596475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.596477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.596722 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.597179 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.597186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.597531 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.597712 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.598064 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":112,"threads_started":1}
I20260929 00:54:01.600548 25600 heartbeater.cc:347] Connected to a master server at 127.11.40.254:37519
I20260929 00:54:01.600857 25600 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:01.601493 25600 heartbeater.cc:511] Master 127.11.40.254:37519 requested a full tablet report, sending...
I20260929 00:54:01.602690 25137 ts_manager.cc:194] Registered new tserver with Master: 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915)
I20260929 00:54:01.603085 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.594063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.11.40.254:37519)
0929 00:54:01.594342 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.594346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.594446 (+ 100us) server_negotiation.cc:407] Connection header received
0929 00:54:01.594607 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.594610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.594665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.594779 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.595702 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.596367 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.597493 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.597761 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.597872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.599518 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.599521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.599524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.599551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.601336 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.601810 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.601813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.601815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.601902 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.602212 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.602216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.602218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.602473 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.602663 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.602813 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.602938 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260929 00:54:01.603426 25137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:38201
I20260929 00:54:01.604940 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:01.609177 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.605769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.11.40.254:37519)
0929 00:54:01.605880 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.605883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.605895 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:54:01.606034 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.606036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.606116 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.606191 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:01.606385 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.606884 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.607786 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.608002 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.608075 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.608338 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:01.608678 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:01.608789 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.608960 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.609025 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 00:54:01.609875 25137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:01.613250 25137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:01.621299 25398 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef9450baf149a4aa6f42b39c9a18a4 (DEFAULT_TABLE table=test-table [id=264a03acdf99479eb37749915769bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:01.621299 25259 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef9450baf149a4aa6f42b39c9a18a4 (DEFAULT_TABLE table=test-table [id=264a03acdf99479eb37749915769bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:01.621971 25259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef9450baf149a4aa6f42b39c9a18a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:01.621971 25398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef9450baf149a4aa6f42b39c9a18a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:01.624939 25356 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Bootstrap starting.
I20260929 00:54:01.624986 25217 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Bootstrap starting.
I20260929 00:54:01.625349 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.620306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:36915 (local address 127.0.0.1:42884)
0929 00:54:01.620471 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.620489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.620576 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.621456 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.621459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.621477 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:01.621605 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.621616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.622781 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.622787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.623714 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.623724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.624910 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.624914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.625021 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.625078 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.625135 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.625222 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 00:54:01.625953 25217 tablet_bootstrap.cc:654] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:01.626013 25356 tablet_bootstrap.cc:654] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:01.626283 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.620484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.11.40.195:36915)
0929 00:54:01.621006 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.621016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.621036 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:54:01.621107 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.621112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.621293 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.621476 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:01.621741 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.622640 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.623844 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.624743 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.625642 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.625947 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.626008 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.626110 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":180,"threads_started":1}
I20260929 00:54:01.627507 25217 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: No bootstrap required, opened a new log
I20260929 00:54:01.627604 25217 ts_tablet_manager.cc:1397] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:01.627705 25356 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: No bootstrap required, opened a new log
I20260929 00:54:01.627791 25356 ts_tablet_manager.cc:1397] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:01.628041 25217 raft_consensus.cc:348] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.628173 25217 raft_consensus.cc:374] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:01.628213 25217 raft_consensus.cc:729] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5507585ce34f38b3a781990e217faa, State: Initialized, Role: FOLLOWER
I20260929 00:54:01.628316 25356 raft_consensus.cc:348] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.628358 25217 consensus_queue.cc:260] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.628458 25356 raft_consensus.cc:374] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:01.628504 25356 raft_consensus.cc:729] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c35f64cf76f40c5ab12a4767e9c69ef, State: Initialized, Role: FOLLOWER
I20260929 00:54:01.628620 25217 ts_tablet_manager.cc:1428] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:54:01.628631 25356 consensus_queue.cc:260] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.628942 25356 ts_tablet_manager.cc:1428] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 00:54:01.629630 25534 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef9450baf149a4aa6f42b39c9a18a4 (DEFAULT_TABLE table=test-table [id=264a03acdf99479eb37749915769bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:01.630538 25534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef9450baf149a4aa6f42b39c9a18a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:01.637642 25612 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Bootstrap starting.
I20260929 00:54:01.639987 25612 tablet_bootstrap.cc:654] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:01.640769 25612 log.cc:824] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:01.642825 25612 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: No bootstrap required, opened a new log
I20260929 00:54:01.642993 25612 ts_tablet_manager.cc:1397] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:54:01.646037 25612 raft_consensus.cc:348] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.646277 25612 raft_consensus.cc:374] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:01.646319 25612 raft_consensus.cc:729] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d62e5553a84e65aa16bbbd8d5169b9, State: Initialized, Role: FOLLOWER
I20260929 00:54:01.646816 25612 consensus_queue.cc:260] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.647571 25600 heartbeater.cc:503] Master 127.11.40.254:37519 was elected leader, sending a full tablet report...
I20260929 00:54:01.647915 25612 ts_tablet_manager.cc:1428] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 00:54:01.660497 25326 tablet.cc:2369] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:01.671352 25614 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:01.671571 25614 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.673089 25614 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
I20260929 00:54:01.678120 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.673903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:38515 (local address 127.11.40.194:43967)
0929 00:54:01.674017 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.674020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.674101 (+ 81us) server_negotiation.cc:407] Connection header received
0929 00:54:01.674180 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.674183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.674248 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.674345 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:01.674663 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.675361 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.676728 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.677434 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.677566 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.677734 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.677883 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.677965 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 00:54:01.678102 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.673451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:43183 (local address 127.11.40.193:45579)
0929 00:54:01.673574 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.673578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.673594 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:01.673706 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.673712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.673777 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.673875 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:01.674395 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.675145 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.676727 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.677434 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.677566 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.677728 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.677883 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.677965 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:54:01.678172 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.673329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.11.40.195:43183)
0929 00:54:01.673487 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.673504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.673613 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.674091 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.674096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.674118 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:01.674252 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.674258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.675251 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.675255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.676549 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.676559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.677639 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.677644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.677757 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.677774 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.677855 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.677992 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":62}
I20260929 00:54:01.678399 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.673586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.11.40.195:38515)
0929 00:54:01.673998 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.674011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.674106 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.674379 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.674381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.674397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:01.674512 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.674518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.675474 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.675477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.676602 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.676612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.677595 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.677599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.677720 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.677753 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.677855 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.677992 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":31,"thread_start_us":162,"threads_started":1}
I20260929 00:54:01.678839 25279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef9450baf149a4aa6f42b39c9a18a4" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5507585ce34f38b3a781990e217faa" is_pre_election: true
I20260929 00:54:01.679139 25418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef9450baf149a4aa6f42b39c9a18a4" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" is_pre_election: true
I20260929 00:54:01.679210 25279 raft_consensus.cc:2454] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 0.
I20260929 00:54:01.679450 25418 raft_consensus.cc:2454] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 0.
I20260929 00:54:01.679785 25487 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d62e5553a84e65aa16bbbd8d5169b9, 1c5507585ce34f38b3a781990e217faa; no voters:
I20260929 00:54:01.680109 25614 raft_consensus.cc:2781] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:01.680194 25614 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:01.680255 25614 raft_consensus.cc:3037] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:01.681432 25614 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.681839 25614 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
I20260929 00:54:01.682297 25279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef9450baf149a4aa6f42b39c9a18a4" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5507585ce34f38b3a781990e217faa"
I20260929 00:54:01.682363 25418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef9450baf149a4aa6f42b39c9a18a4" candidate_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
I20260929 00:54:01.682439 25279 raft_consensus.cc:3037] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:01.682519 25418 raft_consensus.cc:3037] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:01.683527 25279 raft_consensus.cc:2454] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 1.
I20260929 00:54:01.683748 25418 raft_consensus.cc:2454] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d62e5553a84e65aa16bbbd8d5169b9 in term 1.
I20260929 00:54:01.683841 25487 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d62e5553a84e65aa16bbbd8d5169b9, 1c5507585ce34f38b3a781990e217faa; no voters:
I20260929 00:54:01.684015 25614 raft_consensus.cc:2781] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:01.684291 25614 raft_consensus.cc:686] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 LEADER]: Becoming Leader. State: Replica: 09d62e5553a84e65aa16bbbd8d5169b9, State: Running, Role: LEADER
I20260929 00:54:01.684677 25614 consensus_queue.cc:237] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:01.688061 25137 catalog_manager.cc:5495] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:01.711403 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.707759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.11.40.195:36915)
0929 00:54:01.707906 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.707912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.707927 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:54:01.708084 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.708087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.708151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.708279 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:01.708518 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.709074 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.709917 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.710153 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.710266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.710569 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:01.710983 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:01.711118 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.711179 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.711235 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:01.717921 25418 raft_consensus.cc:1261] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Refusing update from remote peer 09d62e5553a84e65aa16bbbd8d5169b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:01.717926 25279 raft_consensus.cc:1261] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Refusing update from remote peer 09d62e5553a84e65aa16bbbd8d5169b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:01.718621 25616 consensus_queue.cc:1035] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:01.718815 25614 consensus_queue.cc:1035] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:01.726040 25619 mvcc.cc:205] Tried to move back new op lower bound from 7334474718068899840 to 7334474717943558144. Current Snapshot: MvccSnapshot[applied={T|T < 7334474718068899840}]
I20260929 00:54:01.728750 25620 mvcc.cc:205] Tried to move back new op lower bound from 7334474718068899840 to 7334474717943558144. Current Snapshot: MvccSnapshot[applied={T|T < 7334474718068899840 or (T in {7334474718068899840})}]
I20260929 00:54:01.734659 25137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:01.734874 25137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:01.737809 25137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Sending DeleteTablet for 3 replicas of tablet 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:01.738665 25398 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef9450baf149a4aa6f42b39c9a18a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:01 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42480
I20260929 00:54:01.738693 25534 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef9450baf149a4aa6f42b39c9a18a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:01 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42884
I20260929 00:54:01.738780 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25175
I20260929 00:54:01.739245 25627 tablet_replica.cc:330] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: stopping tablet replica
I20260929 00:54:01.739409 25471 tablet_replica.cc:330] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: stopping tablet replica
I20260929 00:54:01.739552 25471 raft_consensus.cc:2229] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:01.739670 25627 raft_consensus.cc:2229] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:01.739722 25471 raft_consensus.cc:2258] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:01.740085 25627 raft_consensus.cc:2258] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:01.740892 25471 ts_tablet_manager.cc:1905] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:01.741683 25627 ts_tablet_manager.cc:1905] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:01.743517 25471 ts_tablet_manager.cc:1918] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:01.743589 25471 log.cc:1196] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:01.743841 25471 ts_tablet_manager.cc:1939] T 8cef9450baf149a4aa6f42b39c9a18a4 P 8c35f64cf76f40c5ab12a4767e9c69ef: Deleting consensus metadata
I20260929 00:54:01.744477 25627 ts_tablet_manager.cc:1918] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:01.744618 25627 log.cc:1196] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:01.744949 25121 catalog_manager.cc:4841] TS 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): tablet 8cef9450baf149a4aa6f42b39c9a18a4 (table test-table [id=264a03acdf99479eb37749915769bc3f]) successfully deleted
I20260929 00:54:01.745034 25627 ts_tablet_manager.cc:1939] T 8cef9450baf149a4aa6f42b39c9a18a4 P 09d62e5553a84e65aa16bbbd8d5169b9: Deleting consensus metadata
I20260929 00:54:01.746284 25124 catalog_manager.cc:4841] TS 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): tablet 8cef9450baf149a4aa6f42b39c9a18a4 (table test-table [id=264a03acdf99479eb37749915769bc3f]) successfully deleted
W20260929 00:54:01.748703 25121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:45579 (error 108)
I20260929 00:54:01.749353 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25329
I20260929 00:54:01.750590 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.750267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45579 (local address 127.0.0.1:33134)
0929 00:54:01.750420 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.750495 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:54:01.750840 25121 catalog_manager.cc:4584] TS 1c5507585ce34f38b3a781990e217faa (127.11.40.193:45579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cef9450baf149a4aa6f42b39c9a18a4: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45579: connect: Connection refused (error 111)
I20260929 00:54:01.758889 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25468
I20260929 00:54:01.765103 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25104
I20260929 00:54:01.772047 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--webserver_interface=127.11.40.254
--webserver_port=40801
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:01.889799 25629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.890148 25629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.890239 25629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.894565 25629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:01.894658 25629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.894704 25629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.894759 25629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:01.894797 25629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:01.900370 25629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:01.901635 25629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.902748 25629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:01.911607 25637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.911614 25634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.911609 25635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.911854 25629 server_base.cc:1034] running on GCE node
I20260929 00:54:01.912500 25629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.913902 25629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.915154 25629 hybrid_clock.cc:648] HybridClock initialized: now 1790643241915112 us; error 55 us; skew 500 ppm
Sep 29 00:54:01 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.918732 25629 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:54:01.919872 25629 webserver.cc:460] Webserver started at http://127.11.40.254:40801/ using document root <none> and password file <none>
I20260929 00:54:01.920454 25629 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.920519 25629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.925230 25629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:01.927893 25644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.929662 25629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:01.929829 25629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "6843071baac54c0db74f8f3211bf1739"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "c5809c3c2077781df0244336a776b7f4"
server_key_iv: "f5b4634abfaa351a7568aabfaa2c7c4d"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.930470 25629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.957831 25629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.961220 25629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.961441 25629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.969753 25629 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:37519
I20260929 00:54:01.969781 25696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:37519 every 8 connection(s)
I20260929 00:54:01.971036 25629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:01.975422 25697 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:54:01.977298 25697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap starting.
I20260929 00:54:01.979420 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25629
I20260929 00:54:01.979866 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:45579
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=33897
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:01.995357 25697 log.cc:824] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:02.007000 25697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:02.007442 25697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739: Bootstrap complete.
I20260929 00:54:02.011857 25697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:02.012318 25697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Initialized, Role: FOLLOWER
I20260929 00:54:02.012890 25697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:02.013044 25697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:02.013139 25697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:02.013266 25697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:54:02.015435 25697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:02.015808 25697 leader_election.cc:302] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6843071baac54c0db74f8f3211bf1739; no voters:
I20260929 00:54:02.016184 25697 leader_election.cc:288] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 00:54:02.016278 25701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:54:02.016467 25701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [term 3 LEADER]: Becoming Leader. State: Replica: 6843071baac54c0db74f8f3211bf1739, State: Running, Role: LEADER
I20260929 00:54:02.016791 25701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } }
I20260929 00:54:02.017544 25697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:02.018226 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:54:02.018354 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:02.018677 25703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6843071baac54c0db74f8f3211bf1739. Latest consensus state: current_term: 3 leader_uuid: "6843071baac54c0db74f8f3211bf1739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6843071baac54c0db74f8f3211bf1739" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37519 } } }
I20260929 00:54:02.018819 25703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6843071baac54c0db74f8f3211bf1739 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:02.019109 25708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:02.024469 25708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:02.025473 25708 catalog_manager.cc:1259] Loaded cluster ID: 7db09604ea6b4f2cb3208bb2f5fbc71a
I20260929 00:54:02.025550 25708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 00:54:02.026891 25708 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f014f68c9ad google::LogMessage::Fail() at ??:0
@ 0x7f014f68f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f014f68c4de google::LogMessage::Flush() at ??:0
@ 0x7f014f68da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0159fcbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0159fccd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0159fc9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f015a00d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f01518de3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01518dec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f01518e03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f01518cf512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0152f136db start_thread at ??:0
@ 0x7f014e18a61f clone at ??:0
W20260929 00:54:02.125177 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:02.125568 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:02.125715 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:02.130625 25699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:02.130715 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:02.130784 25699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:54:02.136740 25699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:45579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=33897
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:02.137984 25699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:02.139104 25699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:02.149399 25739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.149398 25738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.149801 25699 server_base.cc:1034] running on GCE node
W20260929 00:54:02.149919 25741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.150337 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:02.151055 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:02.152241 25699 hybrid_clock.cc:648] HybridClock initialized: now 1790643242152210 us; error 49 us; skew 500 ppm
Sep 29 00:54:02 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:02.155126 25699 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:54:02.156133 25699 webserver.cc:460] Webserver started at http://127.11.40.193:33897/ using document root <none> and password file <none>
I20260929 00:54:02.156733 25699 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:02.156796 25699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:02.160924 25699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 00:54:02.163146 25748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.164474 25699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:54:02.164567 25699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "1c5507585ce34f38b3a781990e217faa"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "b9132f06087fbe89b8a492858b045269"
server_key_iv: "6467be26244697e6813a9255a3a02d82"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.165212 25699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:02.191416 25699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:02.195240 25699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:02.195443 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:02.196108 25699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:02.197359 25755 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:54:02.201550 25699 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:54:02.201617 25699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 00:54:02.201664 25699 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:54:02.203392 25699 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:54:02.203449 25699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 00:54:02.203715 25755 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Bootstrap starting.
I20260929 00:54:02.214854 25699 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45579
I20260929 00:54:02.216022 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45579 every 8 connection(s)
I20260929 00:54:02.216197 25699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:02.217399 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25699
I20260929 00:54:02.217852 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:43967
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=32939
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:02.224897 25755 log.cc:824] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:02.235219 25755 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:02.235667 25755 tablet_bootstrap.cc:492] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Bootstrap complete.
I20260929 00:54:02.237138 25755 ts_tablet_manager.cc:1397] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.015s
I20260929 00:54:02.240502 25755 raft_consensus.cc:348] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:02.241103 25755 raft_consensus.cc:729] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5507585ce34f38b3a781990e217faa, State: Initialized, Role: FOLLOWER
I20260929 00:54:02.241676 25755 consensus_queue.cc:260] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:02.242884 25755 ts_tablet_manager.cc:1428] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260929 00:54:02.274559 25865 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:02.218595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:45543)
0929 00:54:02.224865 (+ 6270us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.224945 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:02.226127 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:02.273760 (+ 47633us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5942,"thread_start_us":177,"threads_started":1}
W20260929 00:54:02.275218 25863 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:37519 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:37519: Client connection negotiation failed: client connection to 127.11.40.254:37519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:54:02.275998 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.275701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:38361)
0929 00:54:02.275862 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.275940 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 00:54:02.276710 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.276491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:49109)
0929 00:54:02.276609 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.276652 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:54:02.276965 25863 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:54:02.345381 25866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:02.345638 25866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:02.345728 25866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:02.349867 25866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:02.349927 25866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:02.350030 25866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:54:02.355400 25866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:43967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=32939
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:02.356700 25866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:02.357797 25866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:02.367509 25874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.367511 25873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.367537 25876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.367784 25866 server_base.cc:1034] running on GCE node
I20260929 00:54:02.368359 25866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:02.369153 25866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:02.370352 25866 hybrid_clock.cc:648] HybridClock initialized: now 1790643242370301 us; error 67 us; skew 500 ppm
Sep 29 00:54:02 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:02.373466 25866 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:54:02.374469 25866 webserver.cc:460] Webserver started at http://127.11.40.194:32939/ using document root <none> and password file <none>
I20260929 00:54:02.375041 25866 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:02.375109 25866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:02.379271 25866 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:02.381490 25883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.382771 25866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:54:02.382870 25866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "8c35f64cf76f40c5ab12a4767e9c69ef"
format_stamp: "Formatted at 2026-09-29 00:53:59 on dist-test-slave-5309"
server_key: "375ea74b09c3cc198d8bff768f91f41e"
server_key_iv: "81fd94351418db11bb6bb36f5623ab93"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.383388 25866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:02.415958 25866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:02.419011 25866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:02.419175 25866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:02.419739 25866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:02.420606 25866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:02.420665 25866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.420706 25866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:02.420722 25866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.429728 25866 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:43967
I20260929 00:54:02.429750 25996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:43967 every 8 connection(s)
I20260929 00:54:02.430825 25866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:02.434850 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.433111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:54461)
0929 00:54:02.433643 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.434085 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
W20260929 00:54:02.435343 25997 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:37519 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:37519: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
I20260929 00:54:02.435969 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.435755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:57259)
0929 00:54:02.435868 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.435917 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 00:54:02.436597 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.436404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:55105)
0929 00:54:02.436499 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.436550 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 00:54:02.436898 25997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:54:02.440913 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 25866
I20260929 00:54:02.441313 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:36915
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:37519
--webserver_port=45049
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:02.557363 26000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:02.557680 26000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:02.557757 26000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:02.561810 26000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:02.561923 26000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:02.562023 26000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:54:02.567484 26000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=45049
--tserver_master_addrs=127.11.40.254:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:02.568707 26000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:02.569880 26000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:02.580471 26008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.580526 26000 server_base.cc:1034] running on GCE node
W20260929 00:54:02.580462 26005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.580499 26006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.581223 26000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:02.581938 26000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:02.583124 26000 hybrid_clock.cc:648] HybridClock initialized: now 1790643242583106 us; error 27 us; skew 500 ppm
Sep 29 00:54:02 dist-test-slave-5309 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:02.586123 26000 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:54:02.587208 26000 webserver.cc:460] Webserver started at http://127.11.40.195:45049/ using document root <none> and password file <none>
I20260929 00:54:02.587883 26000 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:02.587990 26000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:02.592085 26000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:02.594573 26015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.596017 26000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:02.596174 26000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "09d62e5553a84e65aa16bbbd8d5169b9"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-5309"
server_key: "14ff40cef7adb3ff58372cb0d4ec22de"
server_key_iv: "aba3c34cb021dba97bb12b39b5df4661"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.596792 26000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:02.629017 26000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:02.632342 26000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:02.632565 26000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:02.633214 26000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:02.634070 26000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:02.634142 26000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.634209 26000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:02.634253 26000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.644088 26000 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:36915
I20260929 00:54:02.644191 26128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:36915 every 8 connection(s)
I20260929 00:54:02.645263 26000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:02.648890 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26000
I20260929 00:54:02.649998 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.647779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:33421)
0929 00:54:02.648386 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.649053 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":136,"threads_started":1}
W20260929 00:54:02.650595 26129 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:37519 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:37519: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
I20260929 00:54:02.651350 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.651068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:43923)
0929 00:54:02.651236 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.651289 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 00:54:02.652172 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.651902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:35345)
0929 00:54:02.652063 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.652111 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 00:54:02.652484 26129 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:54:03.278077 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.277609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:59239)
0929 00:54:03.277883 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:03.277953 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260929 00:54:03.438657 26134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.437918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:60985)
0929 00:54:03.438438 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:03.438523 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":304,"threads_started":1}
I20260929 00:54:03.495918 26135 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:03.496178 26135 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:03.497645 26135 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
Sep 29 00:54:03 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.194@KRBTEST.COM
Sep 29 00:54:03 dist-test-slave-5309 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.195@KRBTEST.COM
I20260929 00:54:03.516667 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.497998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:36915 (local address 127.11.40.193:38591)
0929 00:54:03.498500 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:03.498514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:03.498597 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:03.500625 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:03.500628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:03.501129 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:03.502034 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:03.502057 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.504139 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.504142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:03.505533 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:03.505547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.506676 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.506680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:03.507007 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:03.508048 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:03.508075 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:03.510759 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:03.513996 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:03.513999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:03.514001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:03.514423 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:03.515104 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:03.515107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:03.515108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:03.515191 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:03.515921 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:03.515923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:03.516006 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 00:54:03.516100 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 00:54:03.516440 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":510,"spinlock_wait_cycles":34048,"thread_start_us":73,"threads_started":1}
I20260929 00:54:03.516913 26138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.498285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:52705 (local address 127.11.40.194:43967)
0929 00:54:03.498736 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:03.498755 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:03.498791 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:03.499468 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:03.499473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:03.499840 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:03.500149 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:03.503109 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.504435 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.505714 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.506485 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.506685 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:03.510850 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:03.510859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:03.510882 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:03.510931 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:03.513246 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:03.514050 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:03.514056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:03.514061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:03.514202 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:03.514753 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:03.514759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:03.514761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:03.515370 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:03.515518 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:03.516372 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 00:54:03.516684 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":135,"threads_started":1}
I20260929 00:54:03.516913 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.498308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:38591 (local address 127.11.40.195:36915)
0929 00:54:03.498742 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:03.498755 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:03.498791 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:03.499459 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:03.499465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:03.499841 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:03.500168 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:03.502329 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.503924 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.505714 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.506485 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.506685 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:03.511240 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:03.511256 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:03.511273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:03.511314 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:03.513716 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:03.514655 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:03.514678 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:03.514686 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:03.514870 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:03.515311 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:03.515315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:03.515317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:03.515798 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:03.515904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:03.516372 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 00:54:03.516693 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":107,"threads_started":1}
I20260929 00:54:03.516671 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.497998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:43967 (local address 127.11.40.193:52705)
0929 00:54:03.498312 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:03.498330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:03.498451 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:03.500292 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:03.500298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:03.501129 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:03.502034 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:03.502057 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.504586 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.504590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:03.505533 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:03.505547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:03.506676 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:03.506680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:03.507008 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:03.508085 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:03.508097 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:03.510515 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:03.513422 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:03.513433 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:03.513452 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:03.513890 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:03.514321 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:03.514325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:03.514327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:03.514548 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:03.515561 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:03.515567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:03.515817 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 00:54:03.516053 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:54:03.516494 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":391,"spinlock_wait_cycles":1408}
W20260929 00:54:03.519860 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:03.520241 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:03.520351 25749 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:03.520859 26135 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:03.653766 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:03.653313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:36295)
0929 00:54:03.653600 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:03.653659 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260929 00:54:04.279704 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:04.279082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:59067)
0929 00:54:04.279454 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:04.279578 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260929 00:54:04.440291 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:04.439840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:33263)
0929 00:54:04.440118 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:04.440189 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260929 00:54:04.655330 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:04.654856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:34525)
0929 00:54:04.655132 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:04.655219 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260929 00:54:05.281281 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.280785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:55855)
0929 00:54:05.281118 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:05.281173 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260929 00:54:05.441852 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.441372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:46959)
0929 00:54:05.441670 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:05.441738 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260929 00:54:05.656687 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.656246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:37045)
0929 00:54:05.656532 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:05.656588 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260929 00:54:05.760764 26146 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:05.761015 26146 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:05.761579 26146 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
W20260929 00:54:05.762639 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:05.762775 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:05.762892 25749 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:05.763109 26146 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:06.282471 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.282035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:51987)
0929 00:54:06.282310 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.282370 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260929 00:54:06.443249 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.442750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:51415)
0929 00:54:06.443060 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.443128 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
I20260929 00:54:06.658039 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.657534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:41811)
0929 00:54:06.657850 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.657910 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":141,"threads_started":1}
I20260929 00:54:07.284022 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:07.283486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:49007)
0929 00:54:07.283835 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:07.283908 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":84,"threads_started":1}
I20260929 00:54:07.444801 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:07.444340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:40843)
0929 00:54:07.444626 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:07.444698 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260929 00:54:07.659399 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:07.658923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:49225)
0929 00:54:07.659239 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:07.659301 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20260929 00:54:08.285712 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:08.285174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:54599)
0929 00:54:08.285495 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:08.285608 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
I20260929 00:54:08.446067 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:08.445601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:51551)
0929 00:54:08.445888 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:08.445947 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260929 00:54:08.660938 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:08.660350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:54087)
0929 00:54:08.660696 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:08.660761 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260929 00:54:08.880576 26156 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:08.880739 26156 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:08.881402 26156 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
W20260929 00:54:08.882337 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:08.882474 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:08.882695 25752 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:08.882913 26156 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:09.287053 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.286579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:36067)
0929 00:54:09.286897 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.286954 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260929 00:54:09.447415 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.446912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:60523)
0929 00:54:09.447252 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.447316 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
I20260929 00:54:09.662473 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.661881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:50725)
0929 00:54:09.662313 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.662373 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":128,"threads_started":1}
I20260929 00:54:10.288610 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.288018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:49409)
0929 00:54:10.288358 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.288420 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260929 00:54:10.449100 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.448480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:57099)
0929 00:54:10.448928 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.448995 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":111,"threads_started":1}
I20260929 00:54:10.659184 26162 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:10.659334 26162 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:10.659893 26162 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
W20260929 00:54:10.660888 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:10.661103 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:10.661197 25749 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:10.661384 26162 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:10.663657 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.663228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:38937)
0929 00:54:10.663507 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.663567 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":114,"threads_started":1}
I20260929 00:54:11.290128 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.289578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:50375)
0929 00:54:11.289943 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.290024 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260929 00:54:11.450934 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.450311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:57811)
0929 00:54:11.450718 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.450803 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":116,"threads_started":1}
I20260929 00:54:11.665302 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.664688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:60825)
0929 00:54:11.665124 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.665194 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":242,"threads_started":1}
I20260929 00:54:12.291730 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:12.291214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:41661)
0929 00:54:12.291558 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:12.291627 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
I20260929 00:54:12.452334 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:12.451878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:60599)
0929 00:54:12.452158 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:12.452221 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260929 00:54:12.666713 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:12.666244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:36213)
0929 00:54:12.666546 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:12.666613 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260929 00:54:12.878557 26170 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:12.878756 26170 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:12.879240 26170 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
W20260929 00:54:12.880338 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:12.880470 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:12.880527 25752 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:12.880702 26170 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:13.293330 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:13.292717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:54591)
0929 00:54:13.293136 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:13.293208 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":141,"threads_started":1}
I20260929 00:54:13.454183 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:13.453601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:49615)
0929 00:54:13.453979 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:13.454055 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":187,"threads_started":1}
I20260929 00:54:13.667963 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:13.667512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:38801)
0929 00:54:13.667802 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:13.667861 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260929 00:54:14.295248 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.294615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:55569)
0929 00:54:14.295043 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.295114 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":140,"threads_started":1}
I20260929 00:54:14.455782 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.455194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:42995)
0929 00:54:14.455535 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.455603 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260929 00:54:14.669473 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.668935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:40347)
0929 00:54:14.669264 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.669331 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260929 00:54:15.296644 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.296142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:47377)
0929 00:54:15.296459 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.296536 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260929 00:54:15.342862 26178 raft_consensus.cc:482] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:15.343050 26178 raft_consensus.cc:504] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d62e5553a84e65aa16bbbd8d5169b9" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 36915 } } peers { permanent_uuid: "1c5507585ce34f38b3a781990e217faa" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45579 } } peers { permanent_uuid: "8c35f64cf76f40c5ab12a4767e9c69ef" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 43967 } }
I20260929 00:54:15.343541 26178 leader_election.cc:288] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915), 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967)
W20260929 00:54:15.344347 25752 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09d62e5553a84e65aa16bbbd8d5169b9 (127.11.40.195:36915): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
W20260929 00:54:15.344451 25749 leader_election.cc:341] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c35f64cf76f40c5ab12a4767e9c69ef (127.11.40.194:43967): Not found: Tablet not found: 8cef9450baf149a4aa6f42b39c9a18a4
I20260929 00:54:15.344566 25749 leader_election.cc:302] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c5507585ce34f38b3a781990e217faa; no voters: 09d62e5553a84e65aa16bbbd8d5169b9, 8c35f64cf76f40c5ab12a4767e9c69ef
I20260929 00:54:15.344755 26178 raft_consensus.cc:2726] T 8cef9450baf149a4aa6f42b39c9a18a4 P 1c5507585ce34f38b3a781990e217faa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:54:15.457310 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.456752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:58053)
0929 00:54:15.457145 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.457211 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260929 00:54:15.671159 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.670699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:59155)
0929 00:54:15.670982 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.671048 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260929 00:54:16.298128 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:16.297601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:59073)
0929 00:54:16.297952 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:16.298018 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260929 00:54:16.458803 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:16.458290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:55843)
0929 00:54:16.458632 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:16.458698 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260929 00:54:16.672474 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:16.672029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.195:39475)
0929 00:54:16.672309 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:16.672374 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260929 00:54:17.299829 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:17.299319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.193:36675)
0929 00:54:17.299656 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:17.299724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":77,"threads_started":1}
I20260929 00:54:17.460273 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:17.459830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:37519 (local address 127.11.40.194:57435)
0929 00:54:17.460107 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:17.460167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260929 00:54:17.657480 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25699
I20260929 00:54:17.663679 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 25866
I20260929 00:54:17.669685 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26000
2026-09-29T00:54:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19598 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:17 dist-test-slave-5309 krb5kdc[26188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:17 dist-test-slave-5309 krb5kdc[26188](info): set up 2 sockets
Sep 29 00:54:17 dist-test-slave-5309 krb5kdc[26188](info): commencing operation
krb5kdc: starting...
W20260929 00:54:18.754667 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:54:18 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:54:18.889099 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:19.005590 26201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:19.005879 26201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:19.005923 26201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:19.010140 26201 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:54:19.010211 26201 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:54:19.010227 26201 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:54:19.010241 26201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:19.010260 26201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:19.010279 26201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:19.010337 26201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:19.010360 26201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:19.016016 26201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:19.017318 26201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:19.018508 26201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:54:19.028671 26201 server_base.cc:1034] running on GCE node
W20260929 00:54:19.028671 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:19.028889 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:19.029014 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.029659 26201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:19.030854 26201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:19.032058 26201 hybrid_clock.cc:648] HybridClock initialized: now 1790643259032015 us; error 58 us; skew 500 ppm
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:19.039959 26201 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:54:19.041088 26201 webserver.cc:460] Webserver started at http://127.11.40.254:36627/ using document root <none> and password file <none>
I20260929 00:54:19.041705 26201 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:19.041772 26201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:19.041955 26201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:19.044191 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "641a8420dc9c4d52bcdb78d907efb905"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "f74a412472e9a0a70758d17a59e50e54"
server_key_iv: "df795ceafae3f15586a7df41d71e430f"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.044792 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "641a8420dc9c4d52bcdb78d907efb905"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "f74a412472e9a0a70758d17a59e50e54"
server_key_iv: "df795ceafae3f15586a7df41d71e430f"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.049048 26201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:54:19.051438 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.052726 26201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:19.052896 26201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "641a8420dc9c4d52bcdb78d907efb905"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "f74a412472e9a0a70758d17a59e50e54"
server_key_iv: "df795ceafae3f15586a7df41d71e430f"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.053012 26201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:19.064450 26201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:19.067360 26201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:19.067556 26201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:19.074400 26201 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34111
I20260929 00:54:19.074728 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34111 every 8 connection(s)
I20260929 00:54:19.075379 26201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:19.075973 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26201
I20260929 00:54:19.076128 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:54:19.076440 11427 external_mini_cluster.cc:1428] Setting key dd606b0e58c38a8d2d72fb5073cf247e
I20260929 00:54:19.078536 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:19.085341 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap starting.
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:19.087563 26269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:19.088304 26269 log.cc:824] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:19.090227 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: No bootstrap required, opened a new log
I20260929 00:54:19.091243 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.077949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.11.40.254:34111)
0929 00:54:19.078382 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.078398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.078433 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:54:19.079145 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.079163 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.079507 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.079798 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.080711 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.081787 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.082932 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.083402 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.083519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.086437 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.086441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.086455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.086482 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.088316 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.088854 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.088862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.088869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.088990 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.089291 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.089294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.089295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.089696 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.089903 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.090173 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.090471 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":160,"threads_started":1}
I20260929 00:54:19.094014 26269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:19.094282 26269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:19.094362 26269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Initialized, Role: FOLLOWER
I20260929 00:54:19.094781 26269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:19.094931 26269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:19.095018 26269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:19.095142 26269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:19.096153 26269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:19.096493 26269 leader_election.cc:302] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641a8420dc9c4d52bcdb78d907efb905; no voters:
I20260929 00:54:19.096891 26269 leader_election.cc:288] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:19.097026 26274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:19.097258 26274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 LEADER]: Becoming Leader. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Running, Role: LEADER
I20260929 00:54:19.097627 26274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:19.098189 26269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:19.099346 26276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641a8420dc9c4d52bcdb78d907efb905. Latest consensus state: current_term: 1 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:54:19.099480 26276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:19.099799 26275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:54:19.099897 26275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:19.099900 26283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:19.103142 26283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:19.108534 26283 catalog_manager.cc:1347] Generated new cluster ID: a2379496ad434478b131ff2304352389
I20260929 00:54:19.108619 26283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:19.128654 26283 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:19.129655 26283 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:19.134696 26283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Generated new TSK 0
I20260929 00:54:19.135421 26283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:54:19.178634 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:19.296300 26295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:19.296605 26295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:19.296653 26295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:19.301122 26295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:19.301199 26295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:19.301297 26295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:54:19.308169 26295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:19.309442 26295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:19.310496 26295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:19.320472 26300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:19.320477 26301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.320703 26295 server_base.cc:1034] running on GCE node
W20260929 00:54:19.320551 26303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.321206 26295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:19.321921 26295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:19.323139 26295 hybrid_clock.cc:648] HybridClock initialized: now 1790643259323101 us; error 50 us; skew 500 ppm
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:19.326138 26295 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:54:19.326962 26295 webserver.cc:460] Webserver started at http://127.11.40.193:44951/ using document root <none> and password file <none>
I20260929 00:54:19.327489 26295 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:19.327548 26295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:19.327710 26295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:19.329780 26295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "2f054403f84e49c98f828b9b384dfa60"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "b71b4a69b250213efbdb061b877cd4e6"
server_key_iv: "da53b87d8e9cf217627b971fbb3f90c0"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.330303 26295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "2f054403f84e49c98f828b9b384dfa60"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "b71b4a69b250213efbdb061b877cd4e6"
server_key_iv: "da53b87d8e9cf217627b971fbb3f90c0"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.334053 26295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:54:19.336567 26310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.337937 26295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:19.338044 26295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "2f054403f84e49c98f828b9b384dfa60"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "b71b4a69b250213efbdb061b877cd4e6"
server_key_iv: "da53b87d8e9cf217627b971fbb3f90c0"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.338135 26295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:19.355314 26295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:19.358956 26295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:19.359122 26295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:19.359675 26295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:19.360479 26295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:19.360526 26295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.360569 26295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:19.360584 26295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.370036 26295 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:34447
I20260929 00:54:19.370061 26423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:34447 every 8 connection(s)
I20260929 00:54:19.371012 26295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:19.376317 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26295
I20260929 00:54:19.376477 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:19.376827 11427 external_mini_cluster.cc:1428] Setting key 9d316043987a0b14d1f12c31ad56fecc
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:54:19.388273 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.373175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:60981 (local address 127.11.40.254:34111)
0929 00:54:19.373346 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.373351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.374630 (+ 1279us) server_negotiation.cc:407] Connection header received
0929 00:54:19.375797 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.375800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.375855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.375932 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.377780 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.378540 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.380441 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.380877 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.381004 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.383951 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.383957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.383964 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.384002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.385910 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.386482 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.386488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.386490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.386588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.386941 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.386947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.386949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.387251 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.387361 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.387996 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.388118 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:54:19.389330 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.373674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:60981)
0929 00:54:19.374435 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.374502 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.375364 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.376220 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.376230 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.376833 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:19.377604 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.377630 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.378692 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.378697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.380278 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.380293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.381036 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.381039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.381201 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.381871 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:19.381887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:19.383748 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:19.386044 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.386050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.386065 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.386369 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.386687 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.386689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.386690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.386836 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.387383 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:19.387389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:19.387698 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.387911 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.388215 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":131,"threads_started":1}
I20260929 00:54:19.390849 26424 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:54:19.391124 26424 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:19.391837 26424 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:19.393651 26233 ts_manager.cc:194] Registered new tserver with Master: 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447)
I20260929 00:54:19.394806 26233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:60981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:54:19.409034 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:19.537214 26429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:19.537529 26429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:19.537575 26429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:19.543038 26429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:19.543135 26429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:19.543201 26429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:54:19.549139 26429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:19.550382 26429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:19.551853 26429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:19.564863 26437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:19.564884 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.565105 26429 server_base.cc:1034] running on GCE node
W20260929 00:54:19.564679 26434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.565860 26429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:19.566581 26429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:19.567794 26429 hybrid_clock.cc:648] HybridClock initialized: now 1790643259567720 us; error 86 us; skew 500 ppm
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:19.571692 26429 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:54:19.573359 26429 webserver.cc:460] Webserver started at http://127.11.40.194:36323/ using document root <none> and password file <none>
I20260929 00:54:19.574116 26429 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:19.574196 26429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:19.574433 26429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:19.576690 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "609f04d0c3184d06bc00cc318610819e"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "da57d91405f16a3c98c2d5b266ca4eaa"
server_key_iv: "d583b363d6b14ffb9c5dead22270777d"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.577394 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "609f04d0c3184d06bc00cc318610819e"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "da57d91405f16a3c98c2d5b266ca4eaa"
server_key_iv: "d583b363d6b14ffb9c5dead22270777d"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.581881 26429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:54:19.584798 26444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.586328 26429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:19.586504 26429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "609f04d0c3184d06bc00cc318610819e"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "da57d91405f16a3c98c2d5b266ca4eaa"
server_key_iv: "d583b363d6b14ffb9c5dead22270777d"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.586642 26429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:19.602933 26429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:19.606909 26429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:19.607158 26429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:19.607815 26429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:19.608724 26429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:19.608771 26429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.608877 26429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:19.608918 26429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.619382 26429 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:40565
I20260929 00:54:19.619405 26557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:40565 every 8 connection(s)
I20260929 00:54:19.621059 26429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:19.627094 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26429
I20260929 00:54:19.627206 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:19.627604 11427 external_mini_cluster.cc:1428] Setting key f07df33e2fdb4016b2e8ff984ce06480
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:54:19.639669 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.623500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:42367 (local address 127.11.40.254:34111)
0929 00:54:19.623699 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.623705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.625432 (+ 1727us) server_negotiation.cc:407] Connection header received
0929 00:54:19.626470 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.626474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.626532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.626621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.628282 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.628891 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.630701 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.630931 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.631269 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.634595 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.634598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.634600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.634634 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.636682 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.637900 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.637904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.637905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.638010 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.638436 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.638440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.638442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.638653 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.638770 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.639345 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.639496 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 00:54:19.640616 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.624003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:42367)
0929 00:54:19.625226 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.625279 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.626169 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.626904 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.626911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.627395 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:19.627973 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.627989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.629036 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.629041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.630524 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.630541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.631285 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.631288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.631444 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.632196 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:19.632215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:19.634380 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:19.637120 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.637129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.637149 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.637639 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.638119 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.638122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.638125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.638330 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.638771 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:19.638781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:19.639080 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.639296 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.639684 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":134,"threads_started":1}
I20260929 00:54:19.642647 26558 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
I20260929 00:54:19.643018 26558 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:54:19.643968 26558 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:19.645756 26233 ts_manager.cc:194] Registered new tserver with Master: 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565)
I20260929 00:54:19.646659 26233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:42367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:54:19.674072 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:19.802626 26563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:19.802919 26563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:19.802992 26563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:19.807849 26563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:19.807924 26563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:19.807986 26563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:54:19.813594 26563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:19.814682 26563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:19.815717 26563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:19.825881 26568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:19.825887 26571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.826013 26563 server_base.cc:1034] running on GCE node
W20260929 00:54:19.826203 26569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:19.826639 26563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:19.827306 26563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:19.828716 26563 hybrid_clock.cc:648] HybridClock initialized: now 1790643259828700 us; error 35 us; skew 500 ppm
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:19.831770 26563 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:54:19.832873 26563 webserver.cc:460] Webserver started at http://127.11.40.195:44213/ using document root <none> and password file <none>
I20260929 00:54:19.833482 26563 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:19.833580 26563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:19.833819 26563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:19.835928 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "d3f0b7454d00484cb702752a4b638e26"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "51d350de0c529c4e3ff87db58ee49b9f"
server_key_iv: "0ca87148a7ca8104fbd081e0cb697c04"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.836513 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "d3f0b7454d00484cb702752a4b638e26"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "51d350de0c529c4e3ff87db58ee49b9f"
server_key_iv: "0ca87148a7ca8104fbd081e0cb697c04"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.840306 26563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:54:19.843068 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.844391 26563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:19.844532 26563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "d3f0b7454d00484cb702752a4b638e26"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "51d350de0c529c4e3ff87db58ee49b9f"
server_key_iv: "0ca87148a7ca8104fbd081e0cb697c04"
server_key_version: "encryptionkey@0"
I20260929 00:54:19.844651 26563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:19.875756 26563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:19.879009 26563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:19.879240 26563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:19.879900 26563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:19.880779 26563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:19.880874 26563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.880949 26563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:19.881004 26563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:19.890017 26563 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:37405
I20260929 00:54:19.890090 26691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:37405 every 8 connection(s)
I20260929 00:54:19.891007 26563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:19.893397 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26563
I20260929 00:54:19.893549 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:19.893882 11427 external_mini_cluster.cc:1428] Setting key 7bf97af42678b66415d2579fa4ceb1b5
Sep 29 00:54:19 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643259, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:19.907438 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.893221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:34829 (local address 127.11.40.254:34111)
0929 00:54:19.893390 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.893394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.895031 (+ 1637us) server_negotiation.cc:407] Connection header received
0929 00:54:19.896245 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.896252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.896316 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.896395 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.897954 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.898727 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.900129 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.900471 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.900576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.903322 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.903326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.903328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.903354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.904912 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.905482 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.905486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.905488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.905580 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.905949 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.905951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.905953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.906212 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.906294 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.907124 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.907282 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:19.908255 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.893790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:34829)
0929 00:54:19.894835 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.894897 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.895943 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.896777 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.896785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.897257 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:19.897785 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.897799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.898870 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.898874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.899986 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.899997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.900616 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.900619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.900797 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.901459 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:19.901475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:19.903147 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:19.905071 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.905076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.905092 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.905374 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.905697 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:19.905700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:19.905701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:19.905851 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:19.906314 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:19.906320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:19.906654 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.906888 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.907238 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":161,"threads_started":1}
I20260929 00:54:19.909336 26692 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
I20260929 00:54:19.909540 26692 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:19.909969 26692 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:19.910899 26233 ts_manager.cc:194] Registered new tserver with Master: d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
I20260929 00:54:19.911499 26233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:34829
I20260929 00:54:19.921142 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:19.931241 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.923119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.11.40.254:34111)
0929 00:54:19.923277 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.923282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.923468 (+ 186us) server_negotiation.cc:407] Connection header received
0929 00:54:19.923673 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.923680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.923722 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.923804 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.924527 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.925118 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.926024 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.926233 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.926339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.928044 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.928053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.928056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.928083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.929421 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.929892 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.929897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.929899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.929985 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.930315 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.930318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.930320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.930559 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.930721 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.930949 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.931066 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:54:19.934454 26233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:19.937645 26233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:19.952986 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.946754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.0.0.1:55280)
0929 00:54:19.947064 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.947078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.947155 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.948376 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.948379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.948398 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:19.948500 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.948506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949774 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.949779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.951399 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.951409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952307 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.952436 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952477 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.952602 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.952738 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":344,"thread_start_us":103,"threads_started":1}
I20260929 00:54:19.953378 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.946714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.11.40.194:40565)
0929 00:54:19.947142 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.947158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.947196 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:54:19.947278 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.947283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.947498 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.947738 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:19.948061 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949310 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.951616 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952435 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952592 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952703 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.952934 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.953045 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":121,"threads_started":1}
I20260929 00:54:19.953537 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.947161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.11.40.193:34447)
0929 00:54:19.947930 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.947936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.947960 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:54:19.948035 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.948041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.948244 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.948389 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:19.948597 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949668 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.951537 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952161 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952307 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952482 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.953267 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.953382 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":84,"threads_started":1}
I20260929 00:54:19.953562 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.946264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.11.40.195:37405)
0929 00:54:19.947404 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.947414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.947435 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:54:19.947503 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.947509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.947756 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.947899 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:19.948241 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949384 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.951596 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952435 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952558 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952700 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.953304 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.953411 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":141,"threads_started":1}
I20260929 00:54:19.953208 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.946023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:37405 (local address 127.0.0.1:56110)
0929 00:54:19.946644 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.946695 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.946838 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.947927 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.947930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.947949 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:19.948055 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.948063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949531 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.949533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.951453 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.951459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952764 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.952895 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952918 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.952998 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.953064 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":573,"spinlock_wait_cycles":4352,"thread_start_us":81,"threads_started":1}
I20260929 00:54:19.953253 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.946273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.0.0.1:40642)
0929 00:54:19.946686 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:19.946701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:19.946838 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:19.947730 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:19.947737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:19.947761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:19.947908 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.947916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.949482 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.949485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.951441 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:19.951452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.952631 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.952634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:19.952730 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.952752 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:19.952899 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:54:19.952969 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":193,"spinlock_wait_cycles":10880,"thread_start_us":113,"threads_started":1}
I20260929 00:54:19.957206 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 530932a78d83416380aa6dcb310d60b8 (DEFAULT_TABLE table=test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:19.957206 26626 tablet_service.cc:1467] Processing CreateTablet for tablet 530932a78d83416380aa6dcb310d60b8 (DEFAULT_TABLE table=test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:19.957185 26492 tablet_service.cc:1467] Processing CreateTablet for tablet 530932a78d83416380aa6dcb310d60b8 (DEFAULT_TABLE table=test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:19.958318 26492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530932a78d83416380aa6dcb310d60b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:19.958343 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530932a78d83416380aa6dcb310d60b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:19.958345 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530932a78d83416380aa6dcb310d60b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:19.966528 26709 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Bootstrap starting.
I20260929 00:54:19.966658 26711 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Bootstrap starting.
I20260929 00:54:19.967098 26710 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap starting.
I20260929 00:54:19.969059 26709 tablet_bootstrap.cc:654] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:19.969089 26711 tablet_bootstrap.cc:654] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:19.969234 26710 tablet_bootstrap.cc:654] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:19.969893 26709 log.cc:824] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:19.970202 26710 log.cc:824] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:19.970326 26711 log.cc:824] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:19.972455 26709 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: No bootstrap required, opened a new log
I20260929 00:54:19.972464 26710 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: No bootstrap required, opened a new log
I20260929 00:54:19.972464 26711 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: No bootstrap required, opened a new log
I20260929 00:54:19.972682 26709 ts_tablet_manager.cc:1397] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:54:19.972680 26710 ts_tablet_manager.cc:1397] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:54:19.972680 26711 ts_tablet_manager.cc:1397] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:54:19.975723 26709 raft_consensus.cc:348] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.976001 26709 raft_consensus.cc:374] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:19.976076 26709 raft_consensus.cc:729] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609f04d0c3184d06bc00cc318610819e, State: Initialized, Role: FOLLOWER
I20260929 00:54:19.976569 26709 consensus_queue.cc:260] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.977301 26558 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
I20260929 00:54:19.977180 26711 raft_consensus.cc:348] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.977474 26711 raft_consensus.cc:374] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:19.977540 26711 raft_consensus.cc:729] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0b7454d00484cb702752a4b638e26, State: Initialized, Role: FOLLOWER
I20260929 00:54:19.977707 26709 ts_tablet_manager.cc:1428] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 00:54:19.977768 26710 raft_consensus.cc:348] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.978078 26710 raft_consensus.cc:374] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:19.978057 26711 consensus_queue.cc:260] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.978190 26710 raft_consensus.cc:729] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f054403f84e49c98f828b9b384dfa60, State: Initialized, Role: FOLLOWER
I20260929 00:54:19.978691 26710 consensus_queue.cc:260] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:19.979215 26711 ts_tablet_manager.cc:1428] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 00:54:19.979604 26692 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
I20260929 00:54:19.979714 26710 ts_tablet_manager.cc:1428] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 00:54:19.980083 26424 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
W20260929 00:54:20.122601 26559 tablet.cc:2369] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:20.122725 26425 tablet.cc:2369] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:20.142059 26693 tablet.cc:2369] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:20.292106 26716 raft_consensus.cc:482] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:20.292369 26716 raft_consensus.cc:504] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:20.293669 26716 leader_election.cc:288] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447), 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565)
I20260929 00:54:20.298967 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.294108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:36879 (local address 127.11.40.194:40565)
0929 00:54:20.294263 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:20.294269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:20.294285 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:20.294464 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:20.294472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:20.294541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:20.294628 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:20.294961 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.295777 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.297660 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.298344 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.298450 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:20.298503 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:20.298733 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 00:54:20.298828 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:54:20.298979 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.293993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.11.40.195:36879)
0929 00:54:20.294205 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:20.294222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:20.294328 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:20.294660 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:20.294666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:20.294686 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:20.294819 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:20.294824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.295953 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.295959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:20.297513 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:20.297524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.298450 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.298453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:20.298567 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:20.298615 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 00:54:20.298700 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:54:20.298812 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":140}
I20260929 00:54:20.299130 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.294120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:39685 (local address 127.11.40.193:34447)
0929 00:54:20.294257 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:20.294261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:20.294557 (+ 296us) server_negotiation.cc:407] Connection header received
0929 00:54:20.294761 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:20.294764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:20.294805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:20.294864 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:20.295133 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.295861 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.297660 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.298317 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.298450 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:20.298503 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:20.298921 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 00:54:20.299014 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:54:20.299293 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.293993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.11.40.195:39685)
0929 00:54:20.294439 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:20.294451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:20.294631 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:20.294908 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:20.294914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:20.294932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:20.295028 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:20.295033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.295992 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.295994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:20.297513 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:20.297524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.298575 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.298578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:20.298739 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:20.298763 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:54:20.298874 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:54:20.298942 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":153,"thread_start_us":109,"threads_started":1}
I20260929 00:54:20.299860 26512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530932a78d83416380aa6dcb310d60b8" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609f04d0c3184d06bc00cc318610819e" is_pre_election: true
I20260929 00:54:20.299898 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530932a78d83416380aa6dcb310d60b8" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f054403f84e49c98f828b9b384dfa60" is_pre_election: true
I20260929 00:54:20.300184 26378 raft_consensus.cc:2454] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 0.
I20260929 00:54:20.300192 26512 raft_consensus.cc:2454] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 0.
I20260929 00:54:20.300765 26582 leader_election.cc:302] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60, 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26; no voters:
I20260929 00:54:20.301105 26716 raft_consensus.cc:2781] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:20.301196 26716 raft_consensus.cc:482] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:20.301241 26716 raft_consensus.cc:3037] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:20.302306 26716 raft_consensus.cc:504] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:20.302683 26716 leader_election.cc:288] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 election: Requested vote from peers 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447), 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565)
I20260929 00:54:20.303049 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530932a78d83416380aa6dcb310d60b8" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f054403f84e49c98f828b9b384dfa60"
I20260929 00:54:20.303099 26512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530932a78d83416380aa6dcb310d60b8" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609f04d0c3184d06bc00cc318610819e"
I20260929 00:54:20.303195 26378 raft_consensus.cc:3037] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:20.303220 26512 raft_consensus.cc:3037] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:20.304512 26378 raft_consensus.cc:2454] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 1.
I20260929 00:54:20.304512 26512 raft_consensus.cc:2454] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 1.
I20260929 00:54:20.304878 26581 leader_election.cc:302] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26; no voters:
I20260929 00:54:20.305143 26716 raft_consensus.cc:2781] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:20.305390 26716 raft_consensus.cc:686] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 1 LEADER]: Becoming Leader. State: Replica: d3f0b7454d00484cb702752a4b638e26, State: Running, Role: LEADER
I20260929 00:54:20.305802 26716 consensus_queue.cc:237] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:20.309605 26232 catalog_manager.cc:5495] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3f0b7454d00484cb702752a4b638e26 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "d3f0b7454d00484cb702752a4b638e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:20.322978 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.319006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.11.40.195:37405)
0929 00:54:20.319152 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:20.319157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:20.319172 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:54:20.319344 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:20.319346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:20.319401 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:20.319509 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:20.319776 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.320293 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.321235 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:20.321546 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:20.321675 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:20.322070 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:20.322544 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:20.322691 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:20.322752 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 00:54:20.322810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:20.329247 26378 raft_consensus.cc:1261] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Refusing update from remote peer d3f0b7454d00484cb702752a4b638e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:20.329259 26512 raft_consensus.cc:1261] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Refusing update from remote peer d3f0b7454d00484cb702752a4b638e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:20.330055 26716 consensus_queue.cc:1035] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:20.330284 26719 consensus_queue.cc:1035] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:20.351519 26232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:20.351771 26232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:20.355916 26232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Sending DeleteTablet for 3 replicas of tablet 530932a78d83416380aa6dcb310d60b8
I20260929 00:54:20.357004 26492 tablet_service.cc:1514] Processing DeleteTablet for tablet 530932a78d83416380aa6dcb310d60b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:40642
I20260929 00:54:20.357227 26358 tablet_service.cc:1514] Processing DeleteTablet for tablet 530932a78d83416380aa6dcb310d60b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:55280
I20260929 00:54:20.357687 26731 tablet_replica.cc:330] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: stopping tablet replica
I20260929 00:54:20.357894 26732 tablet_replica.cc:330] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: stopping tablet replica
I20260929 00:54:20.357950 26626 tablet_service.cc:1514] Processing DeleteTablet for tablet 530932a78d83416380aa6dcb310d60b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:56110
I20260929 00:54:20.358191 26732 raft_consensus.cc:2229] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:20.358238 26731 raft_consensus.cc:2229] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:20.358595 26732 raft_consensus.cc:2258] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:20.358594 26731 raft_consensus.cc:2258] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:20.358989 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26295
I20260929 00:54:20.360972 26731 ts_tablet_manager.cc:1905] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:20.364669 26731 ts_tablet_manager.cc:1918] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:20.364897 26731 log.cc:1196] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/530932a78d83416380aa6dcb310d60b8
I20260929 00:54:20.365259 26731 ts_tablet_manager.cc:1939] T 530932a78d83416380aa6dcb310d60b8 P 609f04d0c3184d06bc00cc318610819e: Deleting consensus metadata
I20260929 00:54:20.365417 26733 tablet_replica.cc:330] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: stopping tablet replica
I20260929 00:54:20.365845 26733 raft_consensus.cc:2229] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:20.366454 26733 raft_consensus.cc:2258] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:20.367194 26219 catalog_manager.cc:4841] TS 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565): tablet 530932a78d83416380aa6dcb310d60b8 (table test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]) successfully deleted
I20260929 00:54:20.368337 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26429
W20260929 00:54:20.368426 26220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:34447 (error 108)
I20260929 00:54:20.369402 26733 ts_tablet_manager.cc:1905] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:20.371043 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:20.370652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.0.0.1:55284)
0929 00:54:20.370861 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:20.370936 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260929 00:54:20.371281 26220 catalog_manager.cc:4584] TS 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 530932a78d83416380aa6dcb310d60b8: Network error: Client connection negotiation failed: client connection to 127.11.40.193:34447: connect: Connection refused (error 111)
I20260929 00:54:20.373659 26733 ts_tablet_manager.cc:1918] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:20.373915 26733 log.cc:1196] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/530932a78d83416380aa6dcb310d60b8
I20260929 00:54:20.374238 26733 ts_tablet_manager.cc:1939] T 530932a78d83416380aa6dcb310d60b8 P d3f0b7454d00484cb702752a4b638e26: Deleting consensus metadata
I20260929 00:54:20.375550 26220 catalog_manager.cc:4841] TS d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): tablet 530932a78d83416380aa6dcb310d60b8 (table test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]) successfully deleted
I20260929 00:54:20.376999 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26563
I20260929 00:54:20.383881 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26201
I20260929 00:54:40.392524 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--webserver_interface=127.11.40.254
--webserver_port=36627
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:40.526767 26735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:40.527154 26735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:40.527233 26735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:40.532692 26735 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:54:40.532827 26735 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:54:40.532893 26735 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:54:40.533138 26735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:40.533237 26735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:40.533295 26735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:40.533334 26735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:40.533370 26735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:40.541561 26735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:40.542846 26735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:40.544308 26735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:40.562651 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.562712 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.562714 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:40.562820 26735 server_base.cc:1034] running on GCE node
I20260929 00:54:40.563589 26735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:40.565029 26735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:40.566249 26735 hybrid_clock.cc:648] HybridClock initialized: now 1790643280566228 us; error 36 us; skew 500 ppm
Sep 29 00:54:40 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:40.577487 26735 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:54:40.578904 26735 webserver.cc:460] Webserver started at http://127.11.40.254:36627/ using document root <none> and password file <none>
I20260929 00:54:40.580193 26735 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:40.580310 26735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:40.585592 26735 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:40.588235 26751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:40.590485 26735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:54:40.590647 26735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "641a8420dc9c4d52bcdb78d907efb905"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "f74a412472e9a0a70758d17a59e50e54"
server_key_iv: "df795ceafae3f15586a7df41d71e430f"
server_key_version: "encryptionkey@0"
I20260929 00:54:40.591708 26735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:40.617375 26735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:40.622298 26735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:40.622632 26735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:40.632841 26735 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34111
I20260929 00:54:40.633543 26803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34111 every 8 connection(s)
I20260929 00:54:40.634722 26735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:40.639916 26804 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:54:40.642184 26804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap starting.
I20260929 00:54:40.643293 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26735
I20260929 00:54:40.643788 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:34447
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=44951
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:40.662930 26804 log.cc:824] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:40.675607 26804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:40.676256 26804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap complete.
I20260929 00:54:40.681367 26804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:40.681979 26804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Initialized, Role: FOLLOWER
I20260929 00:54:40.683044 26804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:40.683241 26804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:40.683360 26804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:40.683534 26804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:54:40.686578 26804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:40.687031 26804 leader_election.cc:302] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641a8420dc9c4d52bcdb78d907efb905; no voters:
I20260929 00:54:40.687444 26804 leader_election.cc:288] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:54:40.687738 26808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:54:40.687997 26808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 LEADER]: Becoming Leader. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Running, Role: LEADER
I20260929 00:54:40.688403 26808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:54:40.689368 26804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:40.690274 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:54:40.690583 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:40.691354 26810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641a8420dc9c4d52bcdb78d907efb905. Latest consensus state: current_term: 2 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:54:40.691541 26810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:40.691776 26817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:40.698585 26817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:40.699892 26817 catalog_manager.cc:1259] Loaded cluster ID: a2379496ad434478b131ff2304352389
I20260929 00:54:40.700145 26817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:40.703292 26817 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:40.704136 26817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Loaded TSK: 0
I20260929 00:54:40.719272 26817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Generated new TSK 1
I20260929 00:54:40.719630 26817 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:54:40.794323 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:40.794616 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:40.794664 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:40.799211 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:40.799294 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:40.799402 26806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:54:40.805168 26806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:34447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=44951
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:40.806325 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:40.807447 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:40.818920 26832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.818920 26831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.819003 26834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:40.819355 26806 server_base.cc:1034] running on GCE node
I20260929 00:54:40.819834 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:40.820565 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:40.821758 26806 hybrid_clock.cc:648] HybridClock initialized: now 1790643280821733 us; error 41 us; skew 500 ppm
Sep 29 00:54:40 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:40.824781 26806 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:54:40.825789 26806 webserver.cc:460] Webserver started at http://127.11.40.193:44951/ using document root <none> and password file <none>
I20260929 00:54:40.826407 26806 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:40.826486 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:40.830683 26806 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:40.833472 26841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:40.835064 26806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:40.835246 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "2f054403f84e49c98f828b9b384dfa60"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "b71b4a69b250213efbdb061b877cd4e6"
server_key_iv: "da53b87d8e9cf217627b971fbb3f90c0"
server_key_version: "encryptionkey@0"
I20260929 00:54:40.835865 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:40.849968 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:40.853665 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:40.853879 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:40.854501 26806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:40.855705 26848 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:54:40.860039 26806 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:54:40.860121 26806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 00:54:40.860174 26806 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:54:40.862046 26806 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:54:40.862113 26806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 00:54:40.862474 26848 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap starting.
I20260929 00:54:40.874873 26806 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:34447
I20260929 00:54:40.874910 26955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:34447 every 8 connection(s)
I20260929 00:54:40.876303 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:40.882862 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26806
I20260929 00:54:40.883400 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:40565
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=36323
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:40.885918 26848 log.cc:824] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:54:40 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:40.894930 26848 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:40.895383 26848 tablet_bootstrap.cc:492] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap complete.
I20260929 00:54:40.896696 26848 ts_tablet_manager.cc:1397] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.004s
I20260929 00:54:40.898504 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.878834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:43391 (local address 127.11.40.254:34111)
0929 00:54:40.879285 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:40.879306 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:40.879594 (+ 288us) server_negotiation.cc:407] Connection header received
0929 00:54:40.880849 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:40.880868 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:40.881327 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:40.881982 (+ 655us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:40.883883 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.885441 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.887423 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.888168 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.888372 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.891985 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:40.891992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:40.892009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:40.892048 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:40.894413 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:40.895237 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:40.895246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:40.895256 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:40.895448 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:40.895873 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:40.895877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:40.895879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:40.896398 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:40.896529 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:40.897258 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 00:54:40.897582 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260929 00:54:40.898567 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.878729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:43391)
0929 00:54:40.879402 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:40.879459 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:40.880329 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:40.882234 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:40.882242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:40.882875 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:40.883643 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.883671 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.885665 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.885671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.887253 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.887271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.888360 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.888367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.888628 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.889422 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:40.889450 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:40.891722 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:40.894595 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:40.894605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:40.894622 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:40.895058 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:40.895568 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:40.895572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:40.895574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:40.895757 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:40.896538 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:40.896548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:40.896899 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 00:54:40.897123 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 00:54:40.897486 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":161,"threads_started":1}
I20260929 00:54:40.899842 26848 raft_consensus.cc:348] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:40.900390 26848 raft_consensus.cc:729] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f054403f84e49c98f828b9b384dfa60, State: Initialized, Role: FOLLOWER
I20260929 00:54:40.900930 26848 consensus_queue.cc:260] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } } peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } }
I20260929 00:54:40.901705 26956 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
I20260929 00:54:40.901926 26956 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:40.902015 26848 ts_tablet_manager.cc:1428] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 00:54:40.902458 26956 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:40.904947 26768 ts_manager.cc:194] Registered new tserver with Master: 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447)
I20260929 00:54:40.908321 26768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:43391
I20260929 00:54:40.910692 26956 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
Sep 29 00:54:40 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.193@KRBTEST.COM
I20260929 00:54:40.920845 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.907772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.0.0.1:59266)
0929 00:54:40.908608 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:40.908652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:40.908793 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:40.909363 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:40.909369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:40.911251 (+ 1882us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:40.911394 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.911403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.912696 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.912700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.913670 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.913682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.914403 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.914407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.914518 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.915085 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:40.915100 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:40.916667 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:40.918724 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:40.918734 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:40.918740 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:40.919026 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:40.919356 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:40.919362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:40.919365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:40.919478 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:40.920056 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:40.920064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:40.920207 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 00:54:40.920356 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:54:40.920593 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":119,"threads_started":1}
I20260929 00:54:40.920984 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.907865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.11.40.193:34447)
0929 00:54:40.908328 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:40.908344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:40.908762 (+ 418us) server_negotiation.cc:407] Connection header received
0929 00:54:40.908959 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:40.908965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:40.909212 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:40.909344 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:40.911527 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.912563 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.913814 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.914256 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.914358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.916855 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:40.916873 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:40.916885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:40.916913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:40.918587 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:40.919156 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:40.919167 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:40.919168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:40.919254 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:40.919585 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:40.919589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:40.919591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:40.919854 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:40.920092 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:40.920534 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 00:54:40.920724 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":101,"threads_started":1}
I20260929 00:54:40.922533 26890 tablet_service.cc:1514] Processing DeleteTablet for tablet 530932a78d83416380aa6dcb310d60b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59266
I20260929 00:54:40.923504 26965 tablet_replica.cc:330] stopping tablet replica
I20260929 00:54:40.923836 26965 raft_consensus.cc:2229] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:40.924113 26965 raft_consensus.cc:2258] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:40.925191 26965 ts_tablet_manager.cc:1905] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:40.929445 26965 ts_tablet_manager.cc:1918] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:40.929598 26965 log.cc:1196] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal/wals/530932a78d83416380aa6dcb310d60b8
I20260929 00:54:40.929879 26965 ts_tablet_manager.cc:1939] T 530932a78d83416380aa6dcb310d60b8 P 2f054403f84e49c98f828b9b384dfa60: Deleting consensus metadata
I20260929 00:54:40.932247 26755 catalog_manager.cc:4841] TS 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447): tablet 530932a78d83416380aa6dcb310d60b8 (table test-table [id=16daf5eec1b5465ebec8c1189d9bf74e]) successfully deleted
W20260929 00:54:41.008742 26960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:41.009122 26960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:41.009171 26960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:41.013280 26960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:41.013350 26960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:41.013412 26960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:54:41.018786 26960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:40565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=36323
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:41.019901 26960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:41.021245 26960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:41.034056 26973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.034137 26971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.034056 26970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:41.034332 26960 server_base.cc:1034] running on GCE node
I20260929 00:54:41.035029 26960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:41.035890 26960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:41.037135 26960 hybrid_clock.cc:648] HybridClock initialized: now 1790643281037090 us; error 70 us; skew 500 ppm
Sep 29 00:54:41 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643281, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:41.041265 26960 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:54:41.042375 26960 webserver.cc:460] Webserver started at http://127.11.40.194:36323/ using document root <none> and password file <none>
I20260929 00:54:41.043041 26960 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:41.043124 26960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:41.047211 26960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:41.049381 26980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.050711 26960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:54:41.050838 26960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "609f04d0c3184d06bc00cc318610819e"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "da57d91405f16a3c98c2d5b266ca4eaa"
server_key_iv: "d583b363d6b14ffb9c5dead22270777d"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.051378 26960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:41.068338 26960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:41.073145 26960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:41.073375 26960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:41.074038 26960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:41.074967 26960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:41.075042 26960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.075114 26960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:41.075166 26960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.085093 26960 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:40565
I20260929 00:54:41.085111 27093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:40565 every 8 connection(s)
I20260929 00:54:41.086223 26960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:41.092833 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 26960
I20260929 00:54:41.093357 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:37405
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=44213
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:54:41 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643281, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:41.104017 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.088249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:43317 (local address 127.11.40.254:34111)
0929 00:54:41.088397 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.088401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.089634 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 00:54:41.090693 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.090696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.090760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.090874 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:41.092199 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.093020 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.094454 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.094864 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.094959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.098626 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:41.098630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:41.098632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:41.098662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:41.101151 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.101843 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.101848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.101850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.101958 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.102508 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.102513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.102515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.102754 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:41.102890 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.103535 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.103839 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:54:41.105019 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.088639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:43317)
0929 00:54:41.089466 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.089515 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.090467 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.091076 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.091084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.091475 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:41.092044 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.092061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.093182 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.093185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.094296 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.094307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.095090 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.095097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.095359 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.096299 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:41.096325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:41.098326 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:41.101312 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:41.101321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:41.101349 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:41.101704 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:41.102074 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:41.102078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:41.102080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:41.102260 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:41.102951 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:41.102958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:41.103250 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.103486 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.104027 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":135,"threads_started":1}
I20260929 00:54:41.106667 27094 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
I20260929 00:54:41.106966 27094 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:41.107523 27094 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:41.108716 26768 ts_manager.cc:194] Registered new tserver with Master: 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565)
I20260929 00:54:41.109630 26768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:43317
W20260929 00:54:41.212002 27097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:41.212286 27097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:41.212332 27097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:41.217084 27097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:41.217171 27097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:41.217236 27097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:54:41.223467 27097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:37405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=44213
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:54:41.224574 27097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:41.225664 27097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:41.237159 27105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.237377 27103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:41.237485 27097 server_base.cc:1034] running on GCE node
W20260929 00:54:41.237159 27102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:41.238716 27097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:41.239552 27097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:41.240751 27097 hybrid_clock.cc:648] HybridClock initialized: now 1790643281240741 us; error 67 us; skew 500 ppm
Sep 29 00:54:41 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643281, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:41.244729 27097 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:54:41.245956 27097 webserver.cc:460] Webserver started at http://127.11.40.195:44213/ using document root <none> and password file <none>
I20260929 00:54:41.246657 27097 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:41.246742 27097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:41.251437 27097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:41.253919 27112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.256096 27097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:41.256233 27097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "d3f0b7454d00484cb702752a4b638e26"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "51d350de0c529c4e3ff87db58ee49b9f"
server_key_iv: "0ca87148a7ca8104fbd081e0cb697c04"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.256925 27097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:41.278286 27097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:41.282466 27097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:41.282727 27097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:41.283427 27097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:41.284996 27097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:41.285092 27097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.285179 27097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:41.285225 27097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.297744 27097 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:37405
I20260929 00:54:41.297777 27225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:37405 every 8 connection(s)
I20260929 00:54:41.298975 27097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:41.301718 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27097
Sep 29 00:54:41 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643281, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:54:41.313299 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.302487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.11.40.254:34111)
0929 00:54:41.302761 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.302767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.302960 (+ 193us) server_negotiation.cc:407] Connection header received
0929 00:54:41.303153 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.303156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.303201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.303297 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:41.304146 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.304693 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.306185 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.306544 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.306676 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.308919 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:41.308924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:41.308927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:41.308960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:41.311042 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.311611 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.311614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.311615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.311710 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.312246 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.312251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.312253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.312484 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:41.312725 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.312941 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.313084 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260929 00:54:41.316742 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.301042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52719 (local address 127.11.40.254:34111)
0929 00:54:41.301235 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.301239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.302809 (+ 1570us) server_negotiation.cc:407] Connection header received
0929 00:54:41.303786 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.303789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.303835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.303924 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:41.305134 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.305986 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.307679 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.308035 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.308365 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.312029 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:41.312037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:41.312039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:41.312064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:41.314140 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.314925 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.314928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.314929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.315007 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.315455 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.315459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.315461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.315701 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:41.315794 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.316481 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.316625 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 00:54:41.317648 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.301542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:52719)
0929 00:54:41.302652 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.302708 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.303591 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.304108 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.304115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.304472 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:41.304995 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.305010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.306219 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.306223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.307539 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.307550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.308420 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.308427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.308645 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.309434 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:41.309478 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:41.311691 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:41.314327 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:41.314334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:41.314352 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:41.314764 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:41.315112 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:41.315115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:41.315117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:41.315321 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:41.315851 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:41.315856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:41.316134 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.316300 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.316600 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":722,"threads_started":1}
I20260929 00:54:41.319180 27226 heartbeater.cc:347] Connected to a master server at 127.11.40.254:34111
I20260929 00:54:41.319626 27226 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:41.320261 27226 heartbeater.cc:511] Master 127.11.40.254:34111 requested a full tablet report, sending...
I20260929 00:54:41.321554 26768 ts_manager.cc:194] Registered new tserver with Master: d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
I20260929 00:54:41.322254 26768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:52719
I20260929 00:54:41.327824 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:41.332209 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.328596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.11.40.254:34111)
0929 00:54:41.328729 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.328733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.328935 (+ 202us) server_negotiation.cc:407] Connection header received
0929 00:54:41.329075 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.329078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.329185 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.329279 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:41.329572 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.330114 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.331025 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.331245 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.331328 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.331704 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:41.331851 (+ 147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 00:54:41.332098 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39384: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 00:54:41.332324 26958 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39384: token expired
W20260929 00:54:41.332733 26698 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.40.254:34111)
I20260929 00:54:41.340608 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.333173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.11.40.254:34111)
0929 00:54:41.333296 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.333299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.333315 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:41.333347 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.333349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.333383 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.333446 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:41.334076 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.334588 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.335418 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.335664 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.335753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.337312 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:41.337318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:41.337325 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:41.337349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:41.338924 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.339416 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.339419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.339420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.339530 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:41.339809 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:41.339811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:41.339812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:41.339983 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:41.340087 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.340380 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.340487 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:54:41.343345 26768 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:41.347435 26768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:41.356590 26890 tablet_service.cc:1467] Processing CreateTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 (DEFAULT_TABLE table=test-table [id=64b19907282f4ec09cc3b63b821555b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:41.357465 26890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a938c2aed4440fa8c7a37f664bf92b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:41.360673 26848 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap starting.
I20260929 00:54:41.360916 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.355302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.0.0.1:49748)
0929 00:54:41.355426 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.355443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.355531 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.356563 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.356566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.356630 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:41.356751 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.356759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.358457 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.358460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.359286 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.359295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.360353 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.360356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.360459 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.360520 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.360634 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.360744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 00:54:41.361205 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.355512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.11.40.194:40565)
0929 00:54:41.356021 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.356035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.356063 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:41.356148 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.356154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.356417 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.356583 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:41.356917 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.358331 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.359424 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.360024 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.360329 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.360542 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.360663 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.360779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":150,"threads_started":1}
I20260929 00:54:41.361729 26848 tablet_bootstrap.cc:654] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:41.362663 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.355577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:37405 (local address 127.0.0.1:34024)
0929 00:54:41.356272 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.356286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.356385 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.357137 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.357140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.357167 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:41.357424 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.357432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.358868 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.358875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.359998 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.360009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.362135 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.362139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.362234 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.362257 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.362350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.362438 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":100,"threads_started":1}
I20260929 00:54:41.362682 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.355860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.11.40.195:37405)
0929 00:54:41.356282 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.356295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.356357 (+ 62us) server_negotiation.cc:407] Connection header received
0929 00:54:41.356515 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.356521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.356711 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.356954 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:41.357567 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.358726 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.360488 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.362000 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.362134 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.362295 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.362359 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.362541 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":162,"threads_started":1}
I20260929 00:54:41.363976 26848 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: No bootstrap required, opened a new log
I20260929 00:54:41.364140 26848 ts_tablet_manager.cc:1397] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 00:54:41.364621 26848 raft_consensus.cc:348] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.364753 26848 raft_consensus.cc:374] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:41.364835 26848 raft_consensus.cc:729] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f054403f84e49c98f828b9b384dfa60, State: Initialized, Role: FOLLOWER
I20260929 00:54:41.365013 26848 consensus_queue.cc:260] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.365386 26848 ts_tablet_manager.cc:1428] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 00:54:41.366288 27160 tablet_service.cc:1467] Processing CreateTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 (DEFAULT_TABLE table=test-table [id=64b19907282f4ec09cc3b63b821555b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:41.366292 27028 tablet_service.cc:1467] Processing CreateTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 (DEFAULT_TABLE table=test-table [id=64b19907282f4ec09cc3b63b821555b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:41.367563 27028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a938c2aed4440fa8c7a37f664bf92b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:41.367570 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a938c2aed4440fa8c7a37f664bf92b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:41.376928 27239 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Bootstrap starting.
I20260929 00:54:41.376932 27240 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Bootstrap starting.
W20260929 00:54:41.377765 26957 tablet.cc:2369] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:41.380155 27240 tablet_bootstrap.cc:654] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:41.380685 27239 tablet_bootstrap.cc:654] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:41.381232 27240 log.cc:824] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:41.381598 27239 log.cc:824] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:41.383844 27240 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: No bootstrap required, opened a new log
I20260929 00:54:41.383854 27239 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: No bootstrap required, opened a new log
I20260929 00:54:41.384093 27240 ts_tablet_manager.cc:1397] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:54:41.384090 27239 ts_tablet_manager.cc:1397] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 00:54:41.388340 27239 raft_consensus.cc:348] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.388339 27240 raft_consensus.cc:348] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.388648 27240 raft_consensus.cc:374] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:41.388650 27239 raft_consensus.cc:374] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:41.388703 27240 raft_consensus.cc:729] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609f04d0c3184d06bc00cc318610819e, State: Initialized, Role: FOLLOWER
I20260929 00:54:41.388703 27239 raft_consensus.cc:729] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0b7454d00484cb702752a4b638e26, State: Initialized, Role: FOLLOWER
I20260929 00:54:41.389251 27240 consensus_queue.cc:260] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.389252 27239 consensus_queue.cc:260] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.390484 27239 ts_tablet_manager.cc:1428] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:54:41.390479 27094 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
I20260929 00:54:41.390486 27240 ts_tablet_manager.cc:1428] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 00:54:41.390897 27226 heartbeater.cc:503] Master 127.11.40.254:34111 was elected leader, sending a full tablet report...
W20260929 00:54:41.550124 27227 tablet.cc:2369] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:41.587698 27095 tablet.cc:2369] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:41.610477 27243 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:41.610687 27243 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.611865 27243 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447)
I20260929 00:54:41.616191 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.612194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.11.40.195:43479)
0929 00:54:41.612332 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.612349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.612522 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.612864 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.612867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.612884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:41.613043 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.613051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.614001 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.614005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.614871 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.614879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.615709 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.615835 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.615860 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.615943 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.616029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 00:54:41.616365 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.612297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:36853 (local address 127.11.40.194:40565)
0929 00:54:41.612423 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.612428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.613036 (+ 608us) server_negotiation.cc:407] Connection header received
0929 00:54:41.613097 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.613102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.613168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.613233 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:41.613544 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.614128 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615262 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.615749 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615865 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.615930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.616152 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.616242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:54:41.616401 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.612210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.11.40.195:36853)
0929 00:54:41.612936 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.612947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.613031 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.613270 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.613273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.613290 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:41.613417 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.613424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.614244 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.614247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.615130 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.615140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.615882 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.616015 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.616037 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:41.616114 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 00:54:41.616177 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":371,"thread_start_us":129,"threads_started":1}
I20260929 00:54:41.616217 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.612283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:43479 (local address 127.11.40.193:34447)
0929 00:54:41.612578 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.612581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.612595 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:54:41.612646 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.612648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.612701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.612853 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:41.613175 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.613843 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615058 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.615575 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.615679 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.615794 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.615974 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.616060 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":131,"threads_started":1}
I20260929 00:54:41.617133 27048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a938c2aed4440fa8c7a37f664bf92b7" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609f04d0c3184d06bc00cc318610819e" is_pre_election: true
I20260929 00:54:41.617208 26910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a938c2aed4440fa8c7a37f664bf92b7" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f054403f84e49c98f828b9b384dfa60" is_pre_election: true
I20260929 00:54:41.617489 26910 raft_consensus.cc:2454] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 0.
I20260929 00:54:41.617489 27048 raft_consensus.cc:2454] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 0.
I20260929 00:54:41.618078 27115 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60, 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26; no voters:
I20260929 00:54:41.618412 27243 raft_consensus.cc:2781] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:41.618523 27243 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:41.618587 27243 raft_consensus.cc:3037] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:41.619737 27243 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.620144 27243 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 election: Requested vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447)
I20260929 00:54:41.620702 26910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a938c2aed4440fa8c7a37f664bf92b7" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f054403f84e49c98f828b9b384dfa60"
I20260929 00:54:41.620702 27048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a938c2aed4440fa8c7a37f664bf92b7" candidate_uuid: "d3f0b7454d00484cb702752a4b638e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609f04d0c3184d06bc00cc318610819e"
I20260929 00:54:41.620874 26910 raft_consensus.cc:3037] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:41.620874 27048 raft_consensus.cc:3037] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:41.622254 27048 raft_consensus.cc:2454] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 1.
I20260929 00:54:41.622254 26910 raft_consensus.cc:2454] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f0b7454d00484cb702752a4b638e26 in term 1.
I20260929 00:54:41.622640 27115 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26; no voters:
I20260929 00:54:41.622859 27243 raft_consensus.cc:2781] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:41.623135 27243 raft_consensus.cc:686] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 1 LEADER]: Becoming Leader. State: Replica: d3f0b7454d00484cb702752a4b638e26, State: Running, Role: LEADER
I20260929 00:54:41.623534 27243 consensus_queue.cc:237] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:54:41.627166 26768 catalog_manager.cc:5495] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3f0b7454d00484cb702752a4b638e26 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "d3f0b7454d00484cb702752a4b638e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:41.663009 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.658735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.11.40.195:37405)
0929 00:54:41.658921 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.658925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.658941 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:41.659022 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.659025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.659081 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.659179 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:41.659505 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.660311 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.661407 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.661638 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.661761 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.662107 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:41.662569 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:41.662716 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.662757 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.662818 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:54:41.669095 27048 raft_consensus.cc:1261] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Refusing update from remote peer d3f0b7454d00484cb702752a4b638e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:41.669149 26910 raft_consensus.cc:1261] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Refusing update from remote peer d3f0b7454d00484cb702752a4b638e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:41.669821 27243 consensus_queue.cc:1035] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:41.670075 27247 consensus_queue.cc:1035] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:41.675223 27249 mvcc.cc:205] Tried to move back new op lower bound from 7334474881708957696 to 7334474881532608512. Current Snapshot: MvccSnapshot[applied={T|T < 7334474881708957696}]
I20260929 00:54:41.686622 26768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:41.686847 26768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:41.689661 26768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Sending DeleteTablet for 3 replicas of tablet 3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:54:41.690573 27028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:41 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:49748
I20260929 00:54:41.690739 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26806
I20260929 00:54:41.690712 27160 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:41 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:34024
I20260929 00:54:41.690898 26890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a938c2aed4440fa8c7a37f664bf92b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:41 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:59266
I20260929 00:54:41.692225 27260 tablet_replica.cc:330] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: stopping tablet replica
I20260929 00:54:41.692693 27260 raft_consensus.cc:2229] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:41.692974 27259 tablet_replica.cc:330] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: stopping tablet replica
I20260929 00:54:41.693184 27260 raft_consensus.cc:2258] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:41.693595 27259 raft_consensus.cc:2229] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:41.694031 27259 raft_consensus.cc:2258] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:41.694766 27260 ts_tablet_manager.cc:1905] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:41.695853 27259 ts_tablet_manager.cc:1905] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:41.698472 27260 ts_tablet_manager.cc:1918] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:41.698586 27260 log.cc:1196] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:54:41.698963 27260 ts_tablet_manager.cc:1939] T 3a938c2aed4440fa8c7a37f664bf92b7 P d3f0b7454d00484cb702752a4b638e26: Deleting consensus metadata
I20260929 00:54:41.699225 27259 ts_tablet_manager.cc:1918] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 00:54:41.699323 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:34447 (error 108)
I20260929 00:54:41.699396 27259 log.cc:1196] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:54:41.699891 27259 ts_tablet_manager.cc:1939] T 3a938c2aed4440fa8c7a37f664bf92b7 P 609f04d0c3184d06bc00cc318610819e: Deleting consensus metadata
I20260929 00:54:41.700403 26755 catalog_manager.cc:4841] TS d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): tablet 3a938c2aed4440fa8c7a37f664bf92b7 (table test-table [id=64b19907282f4ec09cc3b63b821555b8]) successfully deleted
I20260929 00:54:41.700577 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26960
I20260929 00:54:41.702805 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.702542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:34447 (local address 127.0.0.1:59270)
0929 00:54:41.702654 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.702729 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 00:54:41.702975 26755 catalog_manager.cc:4584] TS 2f054403f84e49c98f828b9b384dfa60 (127.11.40.193:34447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a938c2aed4440fa8c7a37f664bf92b7: Network error: Client connection negotiation failed: client connection to 127.11.40.193:34447: connect: Connection refused (error 111)
I20260929 00:54:41.709424 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27097
I20260929 00:54:41.711033 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.709699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.0.0.1:49754)
0929 00:54:41.710844 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.710918 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1033}
I20260929 00:54:41.715572 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 26735
I20260929 00:55:01.722821 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--webserver_interface=127.11.40.254
--webserver_port=36627
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:01.846624 27262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:01.847012 27262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:01.847112 27262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:01.851877 27262 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:55:01.851989 27262 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:55:01.852016 27262 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:55:01.852039 27262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:01.852084 27262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:01.852123 27262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:01.852180 27262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:01.852222 27262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:01.858095 27262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:01.859303 27262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:01.860404 27262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:01.870172 27268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:01.870151 27270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:01.870151 27267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:01.870713 27262 server_base.cc:1034] running on GCE node
I20260929 00:55:01.871300 27262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:01.873076 27262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:01.874737 27262 hybrid_clock.cc:648] HybridClock initialized: now 1790643301874679 us; error 67 us; skew 500 ppm
Sep 29 00:55:01 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:01.885782 27262 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:01.886811 27262 webserver.cc:460] Webserver started at http://127.11.40.254:36627/ using document root <none> and password file <none>
I20260929 00:55:01.887384 27262 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:01.887449 27262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:01.891633 27262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:01.894263 27277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:01.895792 27262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:01.895941 27262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "641a8420dc9c4d52bcdb78d907efb905"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "f74a412472e9a0a70758d17a59e50e54"
server_key_iv: "df795ceafae3f15586a7df41d71e430f"
server_key_version: "encryptionkey@0"
I20260929 00:55:01.896543 27262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:01.916226 27262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:01.919804 27262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:01.919973 27262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:01.927762 27262 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34111
I20260929 00:55:01.928109 27329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34111 every 8 connection(s)
I20260929 00:55:01.928966 27262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:01.931015 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27262
I20260929 00:55:01.931497 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:34447
--local_ip_for_outbound_sockets=127.11.40.193
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=44951
--webserver_interface=127.11.40.193
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:55:01.933949 27330 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:55:01.935689 27330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap starting.
I20260929 00:55:01.955942 27330 log.cc:824] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:01.967482 27330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:55:01.967990 27330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Bootstrap complete.
I20260929 00:55:01.972152 27330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:55:01.972609 27330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Initialized, Role: FOLLOWER
I20260929 00:55:01.973349 27330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:55:01.973531 27330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:01.973630 27330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:01.973734 27330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:55:01.975812 27330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:55:01.976181 27330 leader_election.cc:302] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641a8420dc9c4d52bcdb78d907efb905; no voters:
I20260929 00:55:01.976534 27330 leader_election.cc:288] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 00:55:01.976678 27334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:55:01.977032 27334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [term 3 LEADER]: Becoming Leader. State: Replica: 641a8420dc9c4d52bcdb78d907efb905, State: Running, Role: LEADER
I20260929 00:55:01.977367 27334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } }
I20260929 00:55:01.977800 27330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:01.978390 27334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641a8420dc9c4d52bcdb78d907efb905. Latest consensus state: current_term: 3 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:55:01.978516 27334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:01.978946 27335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "641a8420dc9c4d52bcdb78d907efb905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641a8420dc9c4d52bcdb78d907efb905" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34111 } } }
I20260929 00:55:01.979130 27335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:01.979076 27339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:01.984624 27339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:01.985458 27339 catalog_manager.cc:1259] Loaded cluster ID: a2379496ad434478b131ff2304352389
I20260929 00:55:01.985535 27339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:01.988168 27339 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:01.989020 27339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Loaded TSK: 0
I20260929 00:55:01.989111 27339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 641a8420dc9c4d52bcdb78d907efb905: Loaded TSK: 1
F20260929 00:55:01.989910 27339 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f303e4d89ad google::LogMessage::Fail() at ??:0
@ 0x7f303e4db88a google::LogMessage::SendToLog() at ??:0
@ 0x7f303e4d84de google::LogMessage::Flush() at ??:0
@ 0x7f303e4d9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f304119c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3041194f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3048e3d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3048e1799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3048e18178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3048e59df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3048e7882b std::function<>::operator()() at ??:0
@ 0x7f3048e17b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3048e18f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3048e15579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3048e59640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f304072a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f304072ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f304072c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f304071b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3041d5f6db start_thread at ??:0
@ 0x7f303cfd661f clone at ??:0
W20260929 00:55:02.112457 27332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:02.112879 27332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:02.112957 27332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:02.119321 27332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:02.119437 27332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:02.119535 27332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:02.128031 27332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:34447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=44951
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:02.129575 27332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:02.131006 27332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:02.143589 27378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.143620 27377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.143690 27380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.144984 27332 server_base.cc:1034] running on GCE node
I20260929 00:55:02.145488 27332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:02.146201 27332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:02.147430 27332 hybrid_clock.cc:648] HybridClock initialized: now 1790643302147406 us; error 56 us; skew 500 ppm
Sep 29 00:55:02 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:02.150712 27332 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:55:02.151746 27332 webserver.cc:460] Webserver started at http://127.11.40.193:44951/ using document root <none> and password file <none>
I20260929 00:55:02.152350 27332 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:02.152470 27332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:02.156622 27332 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:02.159142 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.160563 27332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:55:02.160722 27332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "2f054403f84e49c98f828b9b384dfa60"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "b71b4a69b250213efbdb061b877cd4e6"
server_key_iv: "da53b87d8e9cf217627b971fbb3f90c0"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.161422 27332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:02.194726 27332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:02.198906 27332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:02.199168 27332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:02.200008 27332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:02.201623 27394 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 00:55:02.206755 27332 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 00:55:02.206843 27332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 00:55:02.206902 27332 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 00:55:02.208750 27332 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 00:55:02.208841 27332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260929 00:55:02.209123 27394 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap starting.
I20260929 00:55:02.221884 27332 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:34447
I20260929 00:55:02.222110 27501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:34447 every 8 connection(s)
I20260929 00:55:02.223766 27332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:02.232772 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27332
I20260929 00:55:02.233538 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:40565
--local_ip_for_outbound_sockets=127.11.40.194
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=36323
--webserver_interface=127.11.40.194
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:55:02.233600 27394 log.cc:824] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:02.244123 27394 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:55:02.244669 27394 tablet_bootstrap.cc:492] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Bootstrap complete.
I20260929 00:55:02.246282 27394 ts_tablet_manager.cc:1397] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.008s
I20260929 00:55:02.250924 27394 raft_consensus.cc:348] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:02.251685 27394 raft_consensus.cc:729] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f054403f84e49c98f828b9b384dfa60, State: Initialized, Role: FOLLOWER
I20260929 00:55:02.252324 27394 consensus_queue.cc:260] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:02.253552 27394 ts_tablet_manager.cc:1428] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
W20260929 00:55:02.272647 27504 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:02.226394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:43011)
0929 00:55:02.227153 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.227221 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:02.228599 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:02.271816 (+ 43217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":210,"threads_started":1}
W20260929 00:55:02.273301 27502 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:34111 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:34111: Client connection negotiation failed: client connection to 127.11.40.254:34111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 00:55:02.274091 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.273816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:53549)
0929 00:55:02.273951 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.274023 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 00:55:02.274857 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.274610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:58513)
0929 00:55:02.274733 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.274791 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:55:02.275065 27502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:55:02.360960 27506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:02.361220 27506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:02.361264 27506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:02.365468 27506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:02.365543 27506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:02.365648 27506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:02.371155 27506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:40565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=36323
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:02.372251 27506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:02.373310 27506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:02.383026 27512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.383042 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.383026 27513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.383419 27506 server_base.cc:1034] running on GCE node
I20260929 00:55:02.383858 27506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:02.384552 27506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:02.385777 27506 hybrid_clock.cc:648] HybridClock initialized: now 1790643302385746 us; error 47 us; skew 500 ppm
Sep 29 00:55:02 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:02.388903 27506 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:55:02.389791 27506 webserver.cc:460] Webserver started at http://127.11.40.194:36323/ using document root <none> and password file <none>
I20260929 00:55:02.390340 27506 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:02.390406 27506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:02.394603 27506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:02.396795 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.398171 27506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:55:02.398370 27506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "609f04d0c3184d06bc00cc318610819e"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "da57d91405f16a3c98c2d5b266ca4eaa"
server_key_iv: "d583b363d6b14ffb9c5dead22270777d"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.398942 27506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:02.412561 27506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:02.416157 27506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:02.416451 27506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:02.417164 27506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:02.418071 27506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:02.418124 27506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.418169 27506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:02.418234 27506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.428908 27506 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:40565
I20260929 00:55:02.428951 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:40565 every 8 connection(s)
I20260929 00:55:02.430016 27506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:02.433990 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.432259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:52217)
0929 00:55:02.433060 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.433183 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":119,"threads_started":1}
W20260929 00:55:02.434507 27636 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:34111 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:34111: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
I20260929 00:55:02.435164 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.434888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:42557)
0929 00:55:02.435053 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.435109 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 00:55:02.435786 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.435597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:46099)
0929 00:55:02.435695 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.435737 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 00:55:02.435966 27636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:55:02.437469 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27506
I20260929 00:55:02.437896 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:37405
--local_ip_for_outbound_sockets=127.11.40.195
--tserver_master_addrs=127.11.40.254:34111
--webserver_port=44213
--webserver_interface=127.11.40.195
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:02.553040 27639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:02.553325 27639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:02.553370 27639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:02.557516 27639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:02.557588 27639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:02.557650 27639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:02.563083 27639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:37405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=44213
--tserver_master_addrs=127.11.40.254:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:02.564239 27639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:02.565397 27639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:02.575829 27644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.575961 27639 server_base.cc:1034] running on GCE node
W20260929 00:55:02.575982 27647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.576160 27645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.576611 27639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:02.577380 27639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:02.578584 27639 hybrid_clock.cc:648] HybridClock initialized: now 1790643302578543 us; error 53 us; skew 500 ppm
Sep 29 00:55:02 dist-test-slave-5309 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:02.581612 27639 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:55:02.582559 27639 webserver.cc:460] Webserver started at http://127.11.40.195:44213/ using document root <none> and password file <none>
I20260929 00:55:02.583155 27639 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:02.583221 27639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:02.587347 27639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:02.589936 27654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.591517 27639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:02.591704 27639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "d3f0b7454d00484cb702752a4b638e26"
format_stamp: "Formatted at 2026-09-29 00:54:19 on dist-test-slave-5309"
server_key: "51d350de0c529c4e3ff87db58ee49b9f"
server_key_iv: "0ca87148a7ca8104fbd081e0cb697c04"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.592401 27639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:02.603924 27639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:02.607141 27639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:02.607366 27639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:02.607995 27639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:02.608932 27639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:02.608999 27639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.609069 27639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:02.609119 27639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.618479 27639 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:37405
I20260929 00:55:02.618496 27767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:37405 every 8 connection(s)
I20260929 00:55:02.619590 27639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:02.623440 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.621878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:35195)
0929 00:55:02.622598 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.622707 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":123,"threads_started":1}
W20260929 00:55:02.624003 27768 heartbeater.cc:644] Failed to heartbeat to 127.11.40.254:34111 (0 consecutive failures): Network error: Failed to ping master at 127.11.40.254:34111: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
I20260929 00:55:02.624706 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.624425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:44025)
0929 00:55:02.624596 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.624651 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 00:55:02.624918 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27639
I20260929 00:55:02.625378 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.625175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:38899)
0929 00:55:02.625273 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.625319 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 00:55:02.625563 27768 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:55:03.276160 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.275712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:47915)
0929 00:55:03.276004 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.276058 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260929 00:55:03.391777 27773 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:03.392067 27773 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:03.393365 27773 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
Sep 29 00:55:03 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.195@KRBTEST.COM
Sep 29 00:55:03 dist-test-slave-5309 krb5kdc[26188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.194@KRBTEST.COM
I20260929 00:55:03.410246 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.393822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:53907 (local address 127.11.40.194:40565)
0929 00:55:03.394285 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.394295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.394326 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:55:03.394977 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.394982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.395345 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.395742 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.397174 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.398759 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.400972 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.401662 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.401847 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.405380 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.405385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.405397 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.405428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.407399 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.408050 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.408057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.408062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.408223 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.408582 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.408584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.408585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.409137 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.409285 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.409814 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.410077 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":111,"threads_started":1}
I20260929 00:55:03.410233 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.393670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:40565 (local address 127.11.40.193:53907)
0929 00:55:03.393881 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.393899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.394013 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.395764 (+ 1751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.395770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.396365 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:03.396953 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.396970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.398991 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.398994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.400678 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.400693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.401815 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.401819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.402177 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.402983 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:03.403015 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:03.405176 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:03.407506 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.407509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.407511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.407889 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.408341 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.408345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.408346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.408441 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.409264 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:03.409271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:03.409501 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.409700 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.410035 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":248,"spinlock_wait_cycles":26496}
I20260929 00:55:03.410969 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.393681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:37405 (local address 127.11.40.193:35943)
0929 00:55:03.393999 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.394026 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.394140 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.395763 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.395769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.396364 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:03.396951 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.396969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.398976 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.398980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.400678 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.400693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.401815 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.401819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.402215 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.403008 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:03.403023 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:03.404866 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:03.407456 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.407466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.407485 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.407850 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.408246 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.408250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.408253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.408435 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.410418 (+ 1983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:03.410422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:03.410521 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.410661 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.410800 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":286,"spinlock_wait_cycles":29696,"thread_start_us":69,"threads_started":1}
I20260929 00:55:03.411267 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.393841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:35943 (local address 127.11.40.195:37405)
0929 00:55:03.394263 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.394276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.394312 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:55:03.394974 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.394980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.395350 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.395718 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.397134 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.398759 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.400882 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.401662 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.401847 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.405181 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.405187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.405204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.405236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.407306 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.407995 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.408001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.408005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.408142 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.408557 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.408560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.408563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.409137 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.409272 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.410805 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.411119 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":79,"threads_started":1}
W20260929 00:55:03.413254 27390 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
W20260929 00:55:03.413559 27391 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:55:03.413656 27391 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60; no voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26
I20260929 00:55:03.414000 27773 raft_consensus.cc:2726] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:55:03.437175 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.436732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:33373)
0929 00:55:03.437024 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.437078 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
I20260929 00:55:03.626785 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.626251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:36563)
0929 00:55:03.626587 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.626665 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260929 00:55:04.277577 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.277088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:38863)
0929 00:55:04.277357 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:04.277408 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260929 00:55:04.438540 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.438006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:52045)
0929 00:55:04.438351 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:04.438416 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260929 00:55:04.628257 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.627804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:41351)
0929 00:55:04.628092 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:04.628156 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260929 00:55:05.152598 27782 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:05.152741 27782 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:05.153213 27782 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
W20260929 00:55:05.154224 27391 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
W20260929 00:55:05.154335 27390 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:55:05.154474 27390 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60; no voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26
I20260929 00:55:05.154620 27782 raft_consensus.cc:2726] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:55:05.279065 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:05.278565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:60657)
0929 00:55:05.278857 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:05.278941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260929 00:55:05.439927 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:05.439477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:44089)
0929 00:55:05.439762 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:05.439831 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260929 00:55:05.629725 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:05.629237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:51893)
0929 00:55:05.629560 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:05.629628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260929 00:55:06.280375 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:06.279958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:58529)
0929 00:55:06.280203 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:06.280262 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260929 00:55:06.441430 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:06.440966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:53679)
0929 00:55:06.441243 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:06.441317 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260929 00:55:06.631102 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:06.630625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:56503)
0929 00:55:06.630918 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:06.630998 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260929 00:55:07.282011 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:07.281512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:49405)
0929 00:55:07.281839 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:07.281905 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260929 00:55:07.443001 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:07.442482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:48219)
0929 00:55:07.442813 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:07.442893 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260929 00:55:07.632609 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:07.632121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:51221)
0929 00:55:07.632440 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:07.632507 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260929 00:55:08.243954 27792 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:08.244128 27792 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:08.244665 27792 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
W20260929 00:55:08.245576 27391 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
W20260929 00:55:08.245684 27390 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:55:08.245780 27390 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60; no voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26
I20260929 00:55:08.245937 27792 raft_consensus.cc:2726] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:55:08.283215 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:08.282787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:42141)
0929 00:55:08.283053 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:08.283108 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260929 00:55:08.444571 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:08.444011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:39049)
0929 00:55:08.444323 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:08.444462 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260929 00:55:08.634012 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:08.633525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:42995)
0929 00:55:08.633831 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:08.633895 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260929 00:55:09.284657 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.284177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:49385)
0929 00:55:09.284488 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:09.284556 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260929 00:55:09.445855 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.445370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:53485)
0929 00:55:09.445695 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:09.445755 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":81,"threads_started":1}
I20260929 00:55:09.635308 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.634838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:55679)
0929 00:55:09.635152 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:09.635210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":155,"threads_started":1}
I20260929 00:55:10.286047 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.285592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:49279)
0929 00:55:10.285890 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:10.285952 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":136,"threads_started":1}
I20260929 00:55:10.447532 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.446853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:55617)
0929 00:55:10.447223 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:10.447317 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260929 00:55:10.636639 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.636173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:50159)
0929 00:55:10.636455 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:10.636543 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260929 00:55:11.287473 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.286993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:54067)
0929 00:55:11.287268 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.287335 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260929 00:55:11.448904 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.448365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:46609)
0929 00:55:11.448700 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.448762 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260929 00:55:11.637828 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.637417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:42757)
0929 00:55:11.637670 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.637730 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260929 00:55:12.288964 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.288472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:38713)
0929 00:55:12.288726 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.288784 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260929 00:55:12.450323 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.449806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:59103)
0929 00:55:12.450135 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.450204 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260929 00:55:12.639249 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.638739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:47495)
0929 00:55:12.639056 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.639117 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
I20260929 00:55:13.093240 27808 raft_consensus.cc:482] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:13.093432 27808 raft_consensus.cc:504] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609f04d0c3184d06bc00cc318610819e" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 40565 } } peers { permanent_uuid: "d3f0b7454d00484cb702752a4b638e26" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 37405 } } peers { permanent_uuid: "2f054403f84e49c98f828b9b384dfa60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 34447 } }
I20260929 00:55:13.093865 27808 leader_election.cc:288] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565), d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405)
W20260929 00:55:13.094666 27390 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 609f04d0c3184d06bc00cc318610819e (127.11.40.194:40565): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
W20260929 00:55:13.094784 27391 leader_election.cc:341] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3f0b7454d00484cb702752a4b638e26 (127.11.40.195:37405): Not found: Tablet not found: 3a938c2aed4440fa8c7a37f664bf92b7
I20260929 00:55:13.094920 27391 leader_election.cc:302] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f054403f84e49c98f828b9b384dfa60; no voters: 609f04d0c3184d06bc00cc318610819e, d3f0b7454d00484cb702752a4b638e26
I20260929 00:55:13.095124 27808 raft_consensus.cc:2726] T 3a938c2aed4440fa8c7a37f664bf92b7 P 2f054403f84e49c98f828b9b384dfa60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 00:55:13.290249 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.289780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:41143)
0929 00:55:13.290080 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.290147 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260929 00:55:13.451611 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.451146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:50969)
0929 00:55:13.451437 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.451499 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260929 00:55:13.640494 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.640080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:40517)
0929 00:55:13.640335 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.640395 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260929 00:55:14.291788 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.291293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:59785)
0929 00:55:14.291619 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.291687 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20260929 00:55:14.453392 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.452600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:38239)
0929 00:55:14.452988 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.453057 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":93,"threads_started":1}
I20260929 00:55:14.642149 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.641647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:36835)
0929 00:55:14.641979 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.642046 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
I20260929 00:55:15.293156 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.292653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:45715)
0929 00:55:15.292981 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.293041 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20260929 00:55:15.454836 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.454384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:48573)
0929 00:55:15.454663 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.454725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260929 00:55:15.643572 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.643145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:51885)
0929 00:55:15.643410 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.643472 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":123,"threads_started":1}
I20260929 00:55:16.294628 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.294148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:39057)
0929 00:55:16.294456 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.294526 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20260929 00:55:16.456166 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.455652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:50501)
0929 00:55:16.456009 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.456063 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":167,"threads_started":1}
I20260929 00:55:16.645090 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.644539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:44721)
0929 00:55:16.644920 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.644990 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260929 00:55:17.296422 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.295852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.193:53853)
0929 00:55:17.296207 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.296312 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260929 00:55:17.457690 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.457230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.194:44815)
0929 00:55:17.457521 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.457588 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260929 00:55:17.637077 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27332
I20260929 00:55:17.643929 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27506
I20260929 00:55:17.646793 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.646237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34111 (local address 127.11.40.195:55875)
0929 00:55:17.646605 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.646670 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.254:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260929 00:55:17.650552 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27639
2026-09-29T00:55:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60786 ms)
[----------] 30 tests from SecurityITest (337745 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:18 dist-test-slave-5309 krb5kdc[27826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:18 dist-test-slave-5309 krb5kdc[27826](info): set up 2 sockets
Sep 29 00:55:18 dist-test-slave-5309 krb5kdc[27826](info): commencing operation
krb5kdc: starting...
W20260929 00:55:19.510141 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:19 dist-test-slave-5309 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:19.642031 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:36961
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:36961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:19.754606 27839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:19.754887 27839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:19.754930 27839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:19.759080 27839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:19.759150 27839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:19.759169 27839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:19.759186 27839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:19.759200 27839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:19.764657 27839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:36961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:36961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:19.765803 27839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:19.766795 27839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:19.776122 27847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:19.776058 27845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:19.776154 27844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:19.776077 27839 server_base.cc:1034] running on GCE node
I20260929 00:55:19.777143 27839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:19.778331 27839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:19.779572 27839 hybrid_clock.cc:648] HybridClock initialized: now 1790643319779539 us; error 42 us; skew 500 ppm
Sep 29 00:55:19 dist-test-slave-5309 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:19.787132 27839 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:19.788398 27839 webserver.cc:460] Webserver started at http://127.11.40.254:43359/ using document root <none> and password file <none>
I20260929 00:55:19.789103 27839 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:19.789206 27839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:19.789471 27839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:19.791800 27839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "8737415213d54946ad48e1bff70b2915"
format_stamp: "Formatted at 2026-09-29 00:55:19 on dist-test-slave-5309"
server_key: "58ac3b230612a8436f261b1f374f8f57"
server_key_iv: "0559ae6402d5af41316ce8629c35a534"
server_key_version: "encryptionkey@0"
I20260929 00:55:19.792459 27839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "8737415213d54946ad48e1bff70b2915"
format_stamp: "Formatted at 2026-09-29 00:55:19 on dist-test-slave-5309"
server_key: "58ac3b230612a8436f261b1f374f8f57"
server_key_iv: "0559ae6402d5af41316ce8629c35a534"
server_key_version: "encryptionkey@0"
I20260929 00:55:19.797740 27839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:55:19.800490 27854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:19.801848 27839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:55:19.802023 27839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "8737415213d54946ad48e1bff70b2915"
format_stamp: "Formatted at 2026-09-29 00:55:19 on dist-test-slave-5309"
server_key: "58ac3b230612a8436f261b1f374f8f57"
server_key_iv: "0559ae6402d5af41316ce8629c35a534"
server_key_version: "encryptionkey@0"
I20260929 00:55:19.802155 27839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:19.819499 27839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:19.822777 27839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:19.823019 27839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:19.830380 27839 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:36961
I20260929 00:55:19.830417 27906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:36961 every 8 connection(s)
I20260929 00:55:19.831439 27839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:19.834331 27907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:19.839233 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27839
I20260929 00:55:19.839401 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:19.839720 11427 external_mini_cluster.cc:1428] Setting key 728611092c388269450c31351d65a57d
I20260929 00:55:19.839851 27907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: Bootstrap starting.
I20260929 00:55:19.842327 27907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:19.843051 27907 log.cc:824] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:19.845016 27907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: No bootstrap required, opened a new log
I20260929 00:55:19.848385 27907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } }
I20260929 00:55:19.848629 27907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:19.848703 27907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8737415213d54946ad48e1bff70b2915, State: Initialized, Role: FOLLOWER
I20260929 00:55:19.849196 27907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } }
Sep 29 00:55:19 dist-test-slave-5309 krb5kdc[27826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:19.849359 27907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:19.849453 27907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:19.849555 27907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:19.850651 27907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } }
I20260929 00:55:19.851044 27907 leader_election.cc:302] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8737415213d54946ad48e1bff70b2915; no voters:
I20260929 00:55:19.851382 27907 leader_election.cc:288] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:19.851511 27912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:19.851758 27912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [term 1 LEADER]: Becoming Leader. State: Replica: 8737415213d54946ad48e1bff70b2915, State: Running, Role: LEADER
I20260929 00:55:19.852164 27912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } }
I20260929 00:55:19.852715 27907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:19.854048 27913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8737415213d54946ad48e1bff70b2915. Latest consensus state: current_term: 1 leader_uuid: "8737415213d54946ad48e1bff70b2915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } } }
I20260929 00:55:19.854190 27913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:19.854809 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8737415213d54946ad48e1bff70b2915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8737415213d54946ad48e1bff70b2915" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 36961 } } }
I20260929 00:55:19.855000 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:19.855257 27922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:19.856410 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:19.841235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.11.40.254:36961)
0929 00:55:19.841663 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:19.841680 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:19.841717 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:55:19.842422 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:19.842474 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:19.842858 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:19.843223 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:19.844121 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.845389 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.846355 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.846904 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.847103 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.850005 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:19.850011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:19.850024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:19.850053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:19.852452 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:19.853020 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:19.853026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:19.853032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:19.853182 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:19.853525 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:19.853530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:19.853535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:19.854099 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:19.854300 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:19.855243 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 00:55:19.855567 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260929 00:55:19.858949 27922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:19.864750 27922 catalog_manager.cc:1347] Generated new cluster ID: a53f329609144a8dae836c521f39dadd
I20260929 00:55:19.864866 27922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:19.875437 27922 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:19.876698 27922 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:19.881865 27922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: Generated new TSK 0
I20260929 00:55:19.882560 27922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:55:19.921386 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:20.052755 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:20.053136 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:20.053184 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:20.058171 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:20.058259 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:20.058326 27933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:20.064170 27933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:20.065435 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:20.066548 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:20.077226 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:20.077255 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.077339 27933 server_base.cc:1034] running on GCE node
W20260929 00:55:20.077244 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.078016 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:20.078771 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:20.079960 27933 hybrid_clock.cc:648] HybridClock initialized: now 1790643320079940 us; error 31 us; skew 500 ppm
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:20.083037 27933 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:55:20.084012 27933 webserver.cc:460] Webserver started at http://127.11.40.193:44897/ using document root <none> and password file <none>
I20260929 00:55:20.084592 27933 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:20.084656 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:20.084970 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:20.087075 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "2ca75297e47747acb674902aec037565"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "cc65f24573f0544b1183d1eeb5e0e741"
server_key_iv: "60a76c35233959cf3d197ea035336cf7"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.087693 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "2ca75297e47747acb674902aec037565"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "cc65f24573f0544b1183d1eeb5e0e741"
server_key_iv: "60a76c35233959cf3d197ea035336cf7"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.091732 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:55:20.094516 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.095954 27933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 00:55:20.096136 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "2ca75297e47747acb674902aec037565"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "cc65f24573f0544b1183d1eeb5e0e741"
server_key_iv: "60a76c35233959cf3d197ea035336cf7"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.096272 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:20.107714 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:20.111290 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:20.111541 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:20.112200 27933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:20.113201 27933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:20.113278 27933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.113364 27933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:20.113418 27933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.123147 27933 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:41223
I20260929 00:55:20.123322 28061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:41223 every 8 connection(s)
I20260929 00:55:20.124358 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:20.129557 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 27933
I20260929 00:55:20.129719 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:20.130038 11427 external_mini_cluster.cc:1428] Setting key e64fd86f59da7e613ba9fbc49fcacd6b
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:55:20.141945 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.126725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:39877 (local address 127.11.40.254:36961)
0929 00:55:20.126873 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.126877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.128314 (+ 1437us) server_negotiation.cc:407] Connection header received
0929 00:55:20.129410 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.129413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.129471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.129556 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.131179 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.131925 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.133978 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.134589 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.134730 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.137759 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:20.137763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:20.137765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:20.137791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:20.139406 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.140128 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.140131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.140133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.140217 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.140661 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.140664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.140665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.140960 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:20.141044 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.141664 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.141789 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:55:20.142940 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.127101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:36961 (local address 127.11.40.193:39877)
0929 00:55:20.128085 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.128133 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.129126 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.129800 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.129809 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.130329 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.131003 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.131020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.132076 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.132080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.133806 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.133821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.134823 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.134826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.135000 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.135689 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.135707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.137542 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:20.139555 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.139562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.139581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.139993 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.140327 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.140330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.140333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.140548 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.141080 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:20.141090 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:20.141382 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.141554 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.141861 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":560,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:55:20.144460 28062 heartbeater.cc:347] Connected to a master server at 127.11.40.254:36961
I20260929 00:55:20.144721 28062 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:20.145411 28062 heartbeater.cc:511] Master 127.11.40.254:36961 requested a full tablet report, sending...
I20260929 00:55:20.147331 27871 ts_manager.cc:194] Registered new tserver with Master: 2ca75297e47747acb674902aec037565 (127.11.40.193:41223)
I20260929 00:55:20.149093 27871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:39877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:55:20.166069 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:20.283049 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:20.283308 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:20.283351 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:20.287461 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:20.287528 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:20.287587 28067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:20.292913 28067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:20.293915 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:20.294899 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:20.304790 28072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:20.304868 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:20.304895 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.305226 28067 server_base.cc:1034] running on GCE node
I20260929 00:55:20.305675 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:20.306332 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:20.307502 28067 hybrid_clock.cc:648] HybridClock initialized: now 1790643320307487 us; error 32 us; skew 500 ppm
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:20.310256 28067 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:55:20.311148 28067 webserver.cc:460] Webserver started at http://127.11.40.194:35809/ using document root <none> and password file <none>
I20260929 00:55:20.311728 28067 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:20.311820 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:20.312054 28067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:20.314195 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "ba10803e749143bf97e00eeac7fb2b11"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "1003ab8e792f655858512924da9a6911"
server_key_iv: "ec51fb5378998d9ec15286ae77190e1c"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.314935 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "ba10803e749143bf97e00eeac7fb2b11"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "1003ab8e792f655858512924da9a6911"
server_key_iv: "ec51fb5378998d9ec15286ae77190e1c"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.319243 28067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:20.322067 28082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.323513 28067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:20.323676 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "ba10803e749143bf97e00eeac7fb2b11"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "1003ab8e792f655858512924da9a6911"
server_key_iv: "ec51fb5378998d9ec15286ae77190e1c"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.323823 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:20.333859 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:20.337234 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:20.337459 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:20.338073 28067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:20.338922 28067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:20.338969 28067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.339039 28067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:20.339075 28067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.348738 28067 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:33509
I20260929 00:55:20.348760 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:33509 every 8 connection(s)
I20260929 00:55:20.349707 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:20.353749 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28067
I20260929 00:55:20.353964 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:20.354266 11427 external_mini_cluster.cc:1428] Setting key 3a2981a453054f72727b030ef0b0433b
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:55:20.365865 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.351633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:39393 (local address 127.11.40.254:36961)
0929 00:55:20.351798 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.351802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.352628 (+ 826us) server_negotiation.cc:407] Connection header received
0929 00:55:20.354481 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.354486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.354556 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.354655 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.356409 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.357027 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.358458 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.358875 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.359009 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.361722 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:20.361730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:20.361734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:20.361770 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:20.363525 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.364089 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.364093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.364095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.364197 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.364608 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.364616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.364617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.364852 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:20.365003 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.365549 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.365688 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:55:20.366824 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.351983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:36961 (local address 127.11.40.194:39393)
0929 00:55:20.352463 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.352507 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.354172 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.354900 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.354912 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.355480 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.356181 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.356206 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.357169 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.357176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.358323 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.358333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.359062 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.359066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.359214 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.359857 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.359873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.361540 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:20.363680 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.363684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.363697 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.363969 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.364299 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.364302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.364303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.364474 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.364998 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:20.365004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:20.365265 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.365479 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.365830 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:55:20.368049 28196 heartbeater.cc:347] Connected to a master server at 127.11.40.254:36961
I20260929 00:55:20.368315 28196 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:20.368882 28196 heartbeater.cc:511] Master 127.11.40.254:36961 requested a full tablet report, sending...
I20260929 00:55:20.370045 27871 ts_manager.cc:194] Registered new tserver with Master: ba10803e749143bf97e00eeac7fb2b11 (127.11.40.194:33509)
I20260929 00:55:20.370695 27871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:39393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:55:20.385975 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:20.501029 28201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:20.501382 28201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:20.501473 28201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:20.505791 28201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:20.505856 28201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:20.505951 28201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:20.511544 28201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:20.512761 28201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:20.513875 28201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:20.523833 28209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:20.523833 28206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.524032 28201 server_base.cc:1034] running on GCE node
W20260929 00:55:20.523833 28207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.524676 28201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:20.525449 28201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:20.526633 28201 hybrid_clock.cc:648] HybridClock initialized: now 1790643320526596 us; error 50 us; skew 500 ppm
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:20.529867 28201 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:55:20.530781 28201 webserver.cc:460] Webserver started at http://127.11.40.195:39599/ using document root <none> and password file <none>
I20260929 00:55:20.531366 28201 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:20.531462 28201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:20.531706 28201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:20.534024 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "9c9cf28d356e4538bdc34408770f6866"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "45035711b1ebca5c50d9ad7f0eee58f7"
server_key_iv: "d0f05b60105a026a0a57f529d8c6c10f"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.534688 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "9c9cf28d356e4538bdc34408770f6866"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "45035711b1ebca5c50d9ad7f0eee58f7"
server_key_iv: "d0f05b60105a026a0a57f529d8c6c10f"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.538671 28201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 00:55:20.541800 28216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.543573 28201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:55:20.543707 28201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "9c9cf28d356e4538bdc34408770f6866"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "45035711b1ebca5c50d9ad7f0eee58f7"
server_key_iv: "d0f05b60105a026a0a57f529d8c6c10f"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.543845 28201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:20.554116 28201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:20.557338 28201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:20.557552 28201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:20.558139 28201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:20.558966 28201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:20.559033 28201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.559103 28201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:20.559154 28201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:20.568347 28201 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:32773
I20260929 00:55:20.568378 28329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:32773 every 8 connection(s)
I20260929 00:55:20.569308 28201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:20.573426 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28201
I20260929 00:55:20.573551 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:20.573899 11427 external_mini_cluster.cc:1428] Setting key 6f297d3b9bc1e0767af3875524c472dd
Sep 29 00:55:20 dist-test-slave-5309 krb5kdc[27826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:20.584105 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.571415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:39265 (local address 127.11.40.254:36961)
0929 00:55:20.571596 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.571601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.572462 (+ 861us) server_negotiation.cc:407] Connection header received
0929 00:55:20.574164 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.574167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.574244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.574340 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.575812 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.576504 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.577736 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.577994 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.578122 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.580662 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:20.580672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:20.580675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:20.580700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:20.582078 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.582599 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.582603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.582605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.582690 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.583030 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.583034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.583036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.583216 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:20.583323 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.583818 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.583941 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:55:20.585093 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.571816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:36961 (local address 127.11.40.195:39265)
0929 00:55:20.572301 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.572344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.573916 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.574536 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.574545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.575080 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.575675 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.575689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.576634 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.576637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.577608 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.577617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.578167 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.578190 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.578378 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.578969 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.578984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.580502 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:20.582205 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.582211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.582226 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.582495 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.582790 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:20.582793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:20.582795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:20.582934 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:20.583331 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:20.583339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:20.583581 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.583778 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.584112 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260929 00:55:20.586445 28330 heartbeater.cc:347] Connected to a master server at 127.11.40.254:36961
I20260929 00:55:20.586709 28330 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:20.587383 28330 heartbeater.cc:511] Master 127.11.40.254:36961 requested a full tablet report, sending...
I20260929 00:55:20.588550 27871 ts_manager.cc:194] Registered new tserver with Master: 9c9cf28d356e4538bdc34408770f6866 (127.11.40.195:32773)
I20260929 00:55:20.589267 27871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:39265
I20260929 00:55:20.600287 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:20.611040 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.602408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.11.40.254:36961)
0929 00:55:20.602605 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.602611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.602793 (+ 182us) server_negotiation.cc:407] Connection header received
0929 00:55:20.602963 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.602966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.603022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.603122 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.603983 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.604528 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.605540 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.605827 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.605950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.607534 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:20.607538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:20.607541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:20.607570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:20.609285 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.609730 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.609734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.609736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.609826 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:20.610117 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:20.610120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:20.610122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:20.610372 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:20.610515 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.610760 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.610876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 00:55:20.614509 27871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:20.617694 27871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:20.633841 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.627036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:33509 (local address 127.0.0.1:39408)
0929 00:55:20.627601 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.627655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.627784 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.628627 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.628633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.628662 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:20.628940 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.628951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.630596 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.630600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.632058 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.632069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633265 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.633389 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.633422 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.633470 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.633646 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20260929 00:55:20.633921 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.628298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.11.40.195:32773)
0929 00:55:20.629357 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.629370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.629391 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:55:20.629467 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.629472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.629632 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.629790 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:20.629983 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.630829 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.632472 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633335 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.633585 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.633681 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.633779 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":183,"threads_started":1}
I20260929 00:55:20.634339 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.627354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.11.40.194:33509)
0929 00:55:20.627997 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.628012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.628040 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:55:20.628122 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.628127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.628434 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.629098 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:20.629252 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.630452 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.632210 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633118 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633940 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.634075 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.634120 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.634202 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":182,"threads_started":1}
I20260929 00:55:20.633844 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.628614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:32773 (local address 127.0.0.1:57414)
0929 00:55:20.628894 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.628906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.628988 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.629764 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.629767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.629785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:20.629887 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.629895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.630992 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.630998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.632333 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.632342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633456 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.633544 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.633571 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.633639 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.633695 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260929 00:55:20.634342 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.628396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.11.40.193:41223)
0929 00:55:20.628888 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.628906 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.628929 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:55:20.629008 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.629016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.629292 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.629727 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:20.629856 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.630855 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.632306 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633034 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633162 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.633320 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.633884 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.634217 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":86,"threads_started":1}
I20260929 00:55:20.633844 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.627891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:41223 (local address 127.0.0.1:39400)
0929 00:55:20.628361 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.628375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.628468 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.629453 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.629457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.629478 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:20.629576 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.629582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.631018 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.631021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.632183 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.632192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.633161 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.633164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.633276 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.633314 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.633462 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.633654 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":75,"thread_start_us":119,"threads_started":1}
I20260929 00:55:20.637267 27996 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 (DEFAULT_TABLE table=test-table [id=cdba8cf3c3d74ead9a3644a6f54d32f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:20.637284 28130 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 (DEFAULT_TABLE table=test-table [id=cdba8cf3c3d74ead9a3644a6f54d32f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:20.637284 28264 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 (DEFAULT_TABLE table=test-table [id=cdba8cf3c3d74ead9a3644a6f54d32f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:20.638252 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9d0685dfa4b5aa47a7964ef95b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:20.638269 28264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9d0685dfa4b5aa47a7964ef95b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:20.638269 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9d0685dfa4b5aa47a7964ef95b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:20.644601 28347 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: Bootstrap starting.
I20260929 00:55:20.645975 28348 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Bootstrap starting.
I20260929 00:55:20.646106 28349 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Bootstrap starting.
I20260929 00:55:20.647833 28347 tablet_bootstrap.cc:654] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:20.648232 28349 tablet_bootstrap.cc:654] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:20.648232 28348 tablet_bootstrap.cc:654] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:20.648733 28347 log.cc:824] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:20.649636 28348 log.cc:824] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:20.649636 28349 log.cc:824] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:20.650944 28347 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: No bootstrap required, opened a new log
I20260929 00:55:20.651450 28347 ts_tablet_manager.cc:1397] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 00:55:20.651616 28348 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: No bootstrap required, opened a new log
I20260929 00:55:20.651616 28349 tablet_bootstrap.cc:492] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: No bootstrap required, opened a new log
I20260929 00:55:20.651808 28348 ts_tablet_manager.cc:1397] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:55:20.651823 28349 ts_tablet_manager.cc:1397] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 00:55:20.655450 28347 raft_consensus.cc:348] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.655720 28347 raft_consensus.cc:374] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:20.655793 28347 raft_consensus.cc:729] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca75297e47747acb674902aec037565, State: Initialized, Role: FOLLOWER
I20260929 00:55:20.655784 28348 raft_consensus.cc:348] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.655784 28349 raft_consensus.cc:348] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.656052 28349 raft_consensus.cc:374] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:20.656122 28349 raft_consensus.cc:729] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba10803e749143bf97e00eeac7fb2b11, State: Initialized, Role: FOLLOWER
I20260929 00:55:20.656159 28348 raft_consensus.cc:374] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:20.656246 28348 raft_consensus.cc:729] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9cf28d356e4538bdc34408770f6866, State: Initialized, Role: FOLLOWER
I20260929 00:55:20.656311 28347 consensus_queue.cc:260] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.656597 28348 consensus_queue.cc:260] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.656597 28349 consensus_queue.cc:260] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.657117 28062 heartbeater.cc:503] Master 127.11.40.254:36961 was elected leader, sending a full tablet report...
I20260929 00:55:20.657516 28347 ts_tablet_manager.cc:1428] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260929 00:55:20.657616 28349 ts_tablet_manager.cc:1428] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 00:55:20.657624 28348 ts_tablet_manager.cc:1428] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:20.657950 28330 heartbeater.cc:503] Master 127.11.40.254:36961 was elected leader, sending a full tablet report...
I20260929 00:55:20.658103 28196 heartbeater.cc:503] Master 127.11.40.254:36961 was elected leader, sending a full tablet report...
I20260929 00:55:20.715086 28355 raft_consensus.cc:482] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:20.715329 28355 raft_consensus.cc:504] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.716518 28355 leader_election.cc:288] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba10803e749143bf97e00eeac7fb2b11 (127.11.40.194:33509), 2ca75297e47747acb674902aec037565 (127.11.40.193:41223)
I20260929 00:55:20.720449 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.716968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:51271 (local address 127.11.40.193:41223)
0929 00:55:20.717101 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.717108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.717388 (+ 280us) server_negotiation.cc:407] Connection header received
0929 00:55:20.717548 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.717550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.717597 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.717674 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:20.717964 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.718468 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.719489 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.719889 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.720017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.720073 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.720254 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.720325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:55:20.720465 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.716781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:41223 (local address 127.11.40.195:51271)
0929 00:55:20.717281 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.717291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.717420 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.717696 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.717699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.717714 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:20.717843 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.717848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.718579 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.718585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.719351 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.719359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.720011 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.720014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.720116 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.720140 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.720237 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.720311 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":157,"thread_start_us":128,"threads_started":1}
I20260929 00:55:20.720616 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.716971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:56575 (local address 127.11.40.194:33509)
0929 00:55:20.717101 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.717108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.717126 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:55:20.717201 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.717203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.717251 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.717375 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:20.717703 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.718223 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.719323 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.719946 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.720080 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.720150 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.720426 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.720500 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:55:20.720616 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.716781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:33509 (local address 127.11.40.195:56575)
0929 00:55:20.716911 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.716926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.717027 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.717395 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.717398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.717432 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:20.717565 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.717571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.718354 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.718359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.719166 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.719176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.720156 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.720161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.720275 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.720293 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:55:20.720394 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:55:20.720475 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":4352}
I20260929 00:55:20.721349 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9d0685dfa4b5aa47a7964ef95b892" candidate_uuid: "9c9cf28d356e4538bdc34408770f6866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca75297e47747acb674902aec037565" is_pre_election: true
I20260929 00:55:20.721381 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9d0685dfa4b5aa47a7964ef95b892" candidate_uuid: "9c9cf28d356e4538bdc34408770f6866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba10803e749143bf97e00eeac7fb2b11" is_pre_election: true
I20260929 00:55:20.721666 28016 raft_consensus.cc:2454] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9cf28d356e4538bdc34408770f6866 in term 0.
I20260929 00:55:20.721676 28150 raft_consensus.cc:2454] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9cf28d356e4538bdc34408770f6866 in term 0.
I20260929 00:55:20.722187 28220 leader_election.cc:302] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca75297e47747acb674902aec037565, 9c9cf28d356e4538bdc34408770f6866; no voters:
I20260929 00:55:20.722453 28355 raft_consensus.cc:2781] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:20.722569 28355 raft_consensus.cc:482] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:20.722630 28355 raft_consensus.cc:3037] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:20.723675 28355 raft_consensus.cc:504] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.724077 28355 leader_election.cc:288] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [CANDIDATE]: Term 1 election: Requested vote from peers ba10803e749143bf97e00eeac7fb2b11 (127.11.40.194:33509), 2ca75297e47747acb674902aec037565 (127.11.40.193:41223)
I20260929 00:55:20.724416 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9d0685dfa4b5aa47a7964ef95b892" candidate_uuid: "9c9cf28d356e4538bdc34408770f6866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba10803e749143bf97e00eeac7fb2b11"
I20260929 00:55:20.724493 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9d0685dfa4b5aa47a7964ef95b892" candidate_uuid: "9c9cf28d356e4538bdc34408770f6866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca75297e47747acb674902aec037565"
I20260929 00:55:20.724558 28150 raft_consensus.cc:3037] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:20.724664 28016 raft_consensus.cc:3037] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:20.725941 28150 raft_consensus.cc:2454] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9cf28d356e4538bdc34408770f6866 in term 1.
I20260929 00:55:20.726066 28016 raft_consensus.cc:2454] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9cf28d356e4538bdc34408770f6866 in term 1.
I20260929 00:55:20.726338 28218 leader_election.cc:302] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c9cf28d356e4538bdc34408770f6866, ba10803e749143bf97e00eeac7fb2b11; no voters:
I20260929 00:55:20.726540 28355 raft_consensus.cc:2781] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:20.726794 28355 raft_consensus.cc:686] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 1 LEADER]: Becoming Leader. State: Replica: 9c9cf28d356e4538bdc34408770f6866, State: Running, Role: LEADER
I20260929 00:55:20.727137 28355 consensus_queue.cc:237] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } }
I20260929 00:55:20.730502 27869 catalog_manager.cc:5495] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c9cf28d356e4538bdc34408770f6866 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "9c9cf28d356e4538bdc34408770f6866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c9cf28d356e4538bdc34408770f6866" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 32773 } health_report { overall_health: HEALTHY } } }
I20260929 00:55:20.759682 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.755893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.11.40.195:32773)
0929 00:55:20.756016 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.756020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.756033 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:55:20.756193 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.756195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.756252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.756347 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:20.756669 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.757253 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.758168 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.758380 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.758465 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.758782 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:20.759211 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:20.759301 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:20.759501 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:55:20.759556 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:55:20.765911 28150 raft_consensus.cc:1261] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 1 FOLLOWER]: Refusing update from remote peer 9c9cf28d356e4538bdc34408770f6866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:20.765911 28016 raft_consensus.cc:1261] T cdf9d0685dfa4b5aa47a7964ef95b892 P 2ca75297e47747acb674902aec037565 [term 1 FOLLOWER]: Refusing update from remote peer 9c9cf28d356e4538bdc34408770f6866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:20.766726 28357 consensus_queue.cc:1035] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba10803e749143bf97e00eeac7fb2b11" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:20.766924 28355 consensus_queue.cc:1035] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca75297e47747acb674902aec037565" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:20.774029 28360 mvcc.cc:205] Tried to move back new op lower bound from 7334475041849008128 to 7334475041701060608. Current Snapshot: MvccSnapshot[applied={T|T < 7334475041849008128}]
I20260929 00:55:20.774698 28361 mvcc.cc:205] Tried to move back new op lower bound from 7334475041849008128 to 7334475041701060608. Current Snapshot: MvccSnapshot[applied={T|T < 7334475041849008128}]
I20260929 00:55:20.782961 27871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:20.783119 27871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:20.785672 27871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8737415213d54946ad48e1bff70b2915: Sending DeleteTablet for 3 replicas of tablet cdf9d0685dfa4b5aa47a7964ef95b892
I20260929 00:55:20.786734 28264 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:57414
I20260929 00:55:20.786744 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39408
I20260929 00:55:20.787264 28369 tablet_replica.cc:330] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: stopping tablet replica
I20260929 00:55:20.787256 27996 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf9d0685dfa4b5aa47a7964ef95b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:20 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:39400
I20260929 00:55:20.787319 28370 tablet_replica.cc:330] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: stopping tablet replica
I20260929 00:55:20.787632 28370 raft_consensus.cc:2229] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:20.787633 28369 raft_consensus.cc:2229] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:20.788005 28369 raft_consensus.cc:2258] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:20.788017 28370 raft_consensus.cc:2258] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:20.788767 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27933
I20260929 00:55:20.789603 28370 ts_tablet_manager.cc:1905] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:20.790251 28369 ts_tablet_manager.cc:1905] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:20.792449 28370 ts_tablet_manager.cc:1918] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:20.792583 28370 log.cc:1196] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/cdf9d0685dfa4b5aa47a7964ef95b892
I20260929 00:55:20.792976 28370 ts_tablet_manager.cc:1939] T cdf9d0685dfa4b5aa47a7964ef95b892 P ba10803e749143bf97e00eeac7fb2b11: Deleting consensus metadata
I20260929 00:55:20.793478 28369 ts_tablet_manager.cc:1918] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:20.793612 28369 log.cc:1196] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/cdf9d0685dfa4b5aa47a7964ef95b892
I20260929 00:55:20.793962 28369 ts_tablet_manager.cc:1939] T cdf9d0685dfa4b5aa47a7964ef95b892 P 9c9cf28d356e4538bdc34408770f6866: Deleting consensus metadata
I20260929 00:55:20.794404 27856 catalog_manager.cc:4841] TS ba10803e749143bf97e00eeac7fb2b11 (127.11.40.194:33509): tablet cdf9d0685dfa4b5aa47a7964ef95b892 (table test-table [id=cdba8cf3c3d74ead9a3644a6f54d32f9]) successfully deleted
I20260929 00:55:20.795495 27855 catalog_manager.cc:4841] TS 9c9cf28d356e4538bdc34408770f6866 (127.11.40.195:32773): tablet cdf9d0685dfa4b5aa47a7964ef95b892 (table test-table [id=cdba8cf3c3d74ead9a3644a6f54d32f9]) successfully deleted
W20260929 00:55:20.799007 27858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:41223 (error 108)
I20260929 00:55:20.799818 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28067
I20260929 00:55:20.800160 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.799835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:41223 (local address 127.0.0.1:39416)
0929 00:55:20.799982 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.800057 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 00:55:20.800611 27858 catalog_manager.cc:4584] TS 2ca75297e47747acb674902aec037565 (127.11.40.193:41223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdf9d0685dfa4b5aa47a7964ef95b892: Network error: Client connection negotiation failed: client connection to 127.11.40.193:41223: connect: Connection refused (error 111)
I20260929 00:55:20.806538 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28201
I20260929 00:55:20.812582 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 27839
2026-09-29T00:55:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2356 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T00:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:20Z Disabled control of system clock
I20260929 00:55:20.844566 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44709
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:20.961102 28376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:20.961374 28376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:20.961418 28376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:20.965487 28376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:20.965549 28376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:20.965565 28376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:20.965582 28376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:20.965597 28376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:20.970801 28376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:44709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:20.971951 28376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:20.973416 28376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:20.982349 28381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:20.982347 28382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.982568 28376 server_base.cc:1034] running on GCE node
W20260929 00:55:20.982380 28384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:20.983314 28376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:20.984607 28376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:20.985803 28376 hybrid_clock.cc:648] HybridClock initialized: now 1790643320985787 us; error 37 us; skew 500 ppm
I20260929 00:55:20.987914 28376 webserver.cc:460] Webserver started at http://127.11.40.254:41869/ using document root <none> and password file <none>
I20260929 00:55:20.988547 28376 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:20.988615 28376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:20.988927 28376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:20.991228 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "e4493d7bebef44d89fc2e7188dda6226"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "f8934b669bc4943f02c54e0f77bdfcff"
server_key_iv: "ce949d46850c9aed218e38603f3421de"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.991854 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "e4493d7bebef44d89fc2e7188dda6226"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "f8934b669bc4943f02c54e0f77bdfcff"
server_key_iv: "ce949d46850c9aed218e38603f3421de"
server_key_version: "encryptionkey@0"
I20260929 00:55:20.995860 28376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:55:20.999119 28390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.000597 28376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:21.000783 28376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "e4493d7bebef44d89fc2e7188dda6226"
format_stamp: "Formatted at 2026-09-29 00:55:20 on dist-test-slave-5309"
server_key: "f8934b669bc4943f02c54e0f77bdfcff"
server_key_iv: "ce949d46850c9aed218e38603f3421de"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.000993 28376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:21.022502 28376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:21.023277 28376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:21.023500 28376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:21.031051 28376 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:44709
I20260929 00:55:21.031061 28442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:44709 every 8 connection(s)
I20260929 00:55:21.032043 28376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:21.035280 28443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:21.042032 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28376
I20260929 00:55:21.042207 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:21.042507 11427 external_mini_cluster.cc:1428] Setting key d2b9614cb1eebe1528ef64255d97d6d5
I20260929 00:55:21.043109 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: Bootstrap starting.
I20260929 00:55:21.046027 28443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:21.046842 28443 log.cc:824] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:21.048986 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: No bootstrap required, opened a new log
I20260929 00:55:21.052680 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.044160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.11.40.254:44709)
0929 00:55:21.045109 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.045124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.045165 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:55:21.045896 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.045916 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.046422 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.046787 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.047255 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.048264 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.049296 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.049782 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.049974 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.050883 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:21.050888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:21.050902 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:21.051295 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:21.051355 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:21.051375 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:21.051491 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.051641 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.051914 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":184,"threads_started":1}
I20260929 00:55:21.053076 28443 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } }
I20260929 00:55:21.053377 28443 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:21.053431 28443 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4493d7bebef44d89fc2e7188dda6226, State: Initialized, Role: FOLLOWER
I20260929 00:55:21.053913 28443 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } }
I20260929 00:55:21.054054 28443 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:21.054137 28443 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:21.054260 28443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:21.055433 28443 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } }
I20260929 00:55:21.055783 28443 leader_election.cc:302] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4493d7bebef44d89fc2e7188dda6226; no voters:
I20260929 00:55:21.056113 28443 leader_election.cc:288] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:21.056289 28448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:21.056582 28448 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [term 1 LEADER]: Becoming Leader. State: Replica: e4493d7bebef44d89fc2e7188dda6226, State: Running, Role: LEADER
I20260929 00:55:21.056984 28448 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } }
I20260929 00:55:21.057540 28443 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:21.058719 28450 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4493d7bebef44d89fc2e7188dda6226. Latest consensus state: current_term: 1 leader_uuid: "e4493d7bebef44d89fc2e7188dda6226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } } }
I20260929 00:55:21.058845 28450 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:21.059146 28449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4493d7bebef44d89fc2e7188dda6226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4493d7bebef44d89fc2e7188dda6226" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44709 } } }
I20260929 00:55:21.059248 28449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:21.059301 28457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:21.062558 28457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:21.068181 28457 catalog_manager.cc:1347] Generated new cluster ID: 71dc4d315d42447fa31ab660f3fd8a80
I20260929 00:55:21.068269 28457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:21.074906 28457 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:21.075874 28457 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:21.083181 28457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: Generated new TSK 0
I20260929 00:55:21.084074 28457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:21.096843 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--builtin_ntp_servers=127.11.40.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:21.213822 28467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:21.214188 28467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:21.214277 28467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:21.218631 28467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:21.218735 28467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:21.218835 28467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:21.223886 28467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:21.225190 28467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:21.226639 28467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:21.236764 28473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:21.236799 28475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:21.236776 28467 server_base.cc:1034] running on GCE node
W20260929 00:55:21.236852 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:21.237461 28467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:21.238160 28467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:21.239387 28467 hybrid_clock.cc:648] HybridClock initialized: now 1790643321239357 us; error 38 us; skew 500 ppm
I20260929 00:55:21.241568 28467 webserver.cc:460] Webserver started at http://127.11.40.193:41225/ using document root <none> and password file <none>
I20260929 00:55:21.242383 28467 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:21.242597 28467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:21.242877 28467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:21.245364 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "87da34b9ee947435ce503ace8ba27c4a"
server_key_iv: "86d5acd500b5218413db7b6af3e17192"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.245961 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "87da34b9ee947435ce503ace8ba27c4a"
server_key_iv: "86d5acd500b5218413db7b6af3e17192"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.249857 28467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:21.252442 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.253849 28467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:21.254026 28467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "87da34b9ee947435ce503ace8ba27c4a"
server_key_iv: "86d5acd500b5218413db7b6af3e17192"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.254149 28467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:21.271374 28467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:21.272177 28467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:21.272403 28467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:21.273151 28467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:21.274219 28467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:21.274293 28467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.274374 28467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:21.274417 28467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.283917 28467 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:35345
I20260929 00:55:21.283962 28594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:35345 every 8 connection(s)
I20260929 00:55:21.284976 28467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:21.294229 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28467
I20260929 00:55:21.294381 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:21.294713 11427 external_mini_cluster.cc:1428] Setting key adf01e93c4be5e1fe47a10e4a1885660
I20260929 00:55:21.296114 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.287109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:55969 (local address 127.11.40.254:44709)
0929 00:55:21.287313 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.287318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.288657 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 00:55:21.290299 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.290303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.290363 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.290506 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.291689 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.292286 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.293753 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.294105 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.294237 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.295573 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:21.295576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:21.295578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:21.295589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:21.295609 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:21.295617 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:21.295679 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.295897 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.295956 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:55:21.297086 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--builtin_ntp_servers=127.11.40.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:21.297166 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.287632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44709 (local address 127.11.40.193:55969)
0929 00:55:21.288383 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.288435 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.289993 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.290824 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.290831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.290923 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.291515 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.291529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.292439 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.292443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.293606 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.293616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.294347 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.294354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.294568 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.295289 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:21.295290 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.295333 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:21.295405 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:21.295705 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:21.295713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:21.295719 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.295858 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.296190 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260929 00:55:21.298596 28595 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44709
I20260929 00:55:21.298839 28595 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:21.299388 28595 heartbeater.cc:511] Master 127.11.40.254:44709 requested a full tablet report, sending...
I20260929 00:55:21.301014 28407 ts_manager.cc:194] Registered new tserver with Master: fb5b09c8f8f8415ea3ac4998b90aa8ce (127.11.40.193:35345)
I20260929 00:55:21.302178 28407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.193:55969
W20260929 00:55:21.413506 28598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:21.413776 28598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:21.413816 28598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:21.417948 28598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:21.418015 28598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:21.418069 28598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:21.422933 28598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:21.424156 28598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:21.425647 28598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:21.436121 28604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:21.436128 28606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:21.436125 28603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:21.436450 28598 server_base.cc:1034] running on GCE node
I20260929 00:55:21.437029 28598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:21.437731 28598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:21.438910 28598 hybrid_clock.cc:648] HybridClock initialized: now 1790643321438871 us; error 48 us; skew 500 ppm
I20260929 00:55:21.441073 28598 webserver.cc:460] Webserver started at http://127.11.40.194:43985/ using document root <none> and password file <none>
I20260929 00:55:21.441699 28598 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:21.441781 28598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:21.442072 28598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:21.444279 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "0d4e17dcf8fd44658f4892da23426dad"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "2a2b6b2616cc587e4ec593d94ba02ec9"
server_key_iv: "4c1de77aa6c17500e36f5bce1f0f32a8"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.444965 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "0d4e17dcf8fd44658f4892da23426dad"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "2a2b6b2616cc587e4ec593d94ba02ec9"
server_key_iv: "4c1de77aa6c17500e36f5bce1f0f32a8"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.449087 28598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:21.451859 28612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.453384 28598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:21.453548 28598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "0d4e17dcf8fd44658f4892da23426dad"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "2a2b6b2616cc587e4ec593d94ba02ec9"
server_key_iv: "4c1de77aa6c17500e36f5bce1f0f32a8"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.453675 28598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:21.469931 28598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:21.470715 28598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:21.470927 28598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:21.471562 28598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:21.472556 28598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:21.472630 28598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.472705 28598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:21.472750 28598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.481544 28598 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:35667
I20260929 00:55:21.481565 28725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:35667 every 8 connection(s)
I20260929 00:55:21.482499 28598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:21.484534 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28598
I20260929 00:55:21.484675 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:21.485051 11427 external_mini_cluster.cc:1428] Setting key 0001410c3ce6725464efb9f3618a04e3
I20260929 00:55:21.486974 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--builtin_ntp_servers=127.11.40.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:21.493582 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.484534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:35571 (local address 127.11.40.254:44709)
0929 00:55:21.484709 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.484716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.485921 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 00:55:21.487388 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.487391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.487452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.487533 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.488671 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.489405 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.491279 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.491537 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.491630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.493085 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:21.493088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:21.493090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:21.493103 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:21.493121 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:21.493128 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:21.493190 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.493409 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.493464 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:55:21.495114 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.484971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44709 (local address 127.11.40.194:35571)
0929 00:55:21.485749 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.485803 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.487165 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.487704 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.487710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.487796 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.488482 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.488504 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.489550 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.489557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.491148 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.491162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.491725 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.491732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.491996 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.492705 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:21.492709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.492777 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:21.492911 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:21.493223 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:21.493235 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:21.493242 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.493397 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.493886 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260929 00:55:21.496551 28726 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44709
I20260929 00:55:21.496888 28726 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:21.497473 28726 heartbeater.cc:511] Master 127.11.40.254:44709 requested a full tablet report, sending...
I20260929 00:55:21.498656 28407 ts_manager.cc:194] Registered new tserver with Master: 0d4e17dcf8fd44658f4892da23426dad (127.11.40.194:35667)
I20260929 00:55:21.499302 28407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.194:35571
W20260929 00:55:21.607321 28729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:21.607653 28729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:21.607723 28729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:21.611932 28729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:21.612032 28729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:21.612133 28729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:21.617653 28729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:21.618922 28729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:21.620397 28729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:21.630645 28734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:21.630620 28729 server_base.cc:1034] running on GCE node
W20260929 00:55:21.630615 28735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:21.630643 28737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:21.631325 28729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:21.632061 28729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:21.633325 28729 hybrid_clock.cc:648] HybridClock initialized: now 1790643321633262 us; error 72 us; skew 500 ppm
I20260929 00:55:21.635382 28729 webserver.cc:460] Webserver started at http://127.11.40.195:33605/ using document root <none> and password file <none>
I20260929 00:55:21.635980 28729 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:21.636075 28729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:21.636327 28729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:21.638494 28729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "66cb44bcc45a4ac787b3f82d31159057"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "e0de47de912c628b15601d3971f3ef59"
server_key_iv: "c1763883fbedab32db85d05b4c17f2e2"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.639065 28729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "66cb44bcc45a4ac787b3f82d31159057"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "e0de47de912c628b15601d3971f3ef59"
server_key_iv: "c1763883fbedab32db85d05b4c17f2e2"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.643028 28729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:21.645835 28743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.647305 28729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:21.647467 28729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "66cb44bcc45a4ac787b3f82d31159057"
format_stamp: "Formatted at 2026-09-29 00:55:21 on dist-test-slave-5309"
server_key: "e0de47de912c628b15601d3971f3ef59"
server_key_iv: "c1763883fbedab32db85d05b4c17f2e2"
server_key_version: "encryptionkey@0"
I20260929 00:55:21.647603 28729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:21.665643 28729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:21.666424 28729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:21.666649 28729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:21.667301 28729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:21.668370 28729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:21.668455 28729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.668531 28729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:21.668603 28729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:21.678318 28729 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:41187
I20260929 00:55:21.678344 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:41187 every 8 connection(s)
I20260929 00:55:21.679303 28729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:21.684659 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28729
I20260929 00:55:21.684782 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:21.685303 11427 external_mini_cluster.cc:1428] Setting key caf46df4bb0648a13f4a37135bd9c573
I20260929 00:55:21.689841 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.681348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:59843 (local address 127.11.40.254:44709)
0929 00:55:21.681533 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.681538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.682759 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 00:55:21.684280 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.684284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.684377 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.684492 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.685760 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.686358 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.687788 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.688049 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.688136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.689338 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:21.689341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:21.689343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:21.689357 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:21.689385 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:21.689393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:21.689458 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.689637 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.689709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 00:55:21.690773 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.681831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44709 (local address 127.11.40.195:59843)
0929 00:55:21.682597 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.682642 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.683959 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.684800 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.684814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.684974 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.685563 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.685580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.686538 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.686541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.687672 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.687681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.688192 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.688194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.688348 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.689073 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:21.689074 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.689118 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:21.689188 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:21.689472 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:21.689478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:21.689483 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.689583 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.689931 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260929 00:55:21.692018 28857 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44709
I20260929 00:55:21.692246 28857 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:21.692759 28857 heartbeater.cc:511] Master 127.11.40.254:44709 requested a full tablet report, sending...
I20260929 00:55:21.693853 28407 ts_manager.cc:194] Registered new tserver with Master: 66cb44bcc45a4ac787b3f82d31159057 (127.11.40.195:41187)
I20260929 00:55:21.694445 28407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.195:59843
I20260929 00:55:21.700039 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:21.706298 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.702282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.11.40.254:44709)
0929 00:55:21.702424 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.702434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.702608 (+ 174us) server_negotiation.cc:407] Connection header received
0929 00:55:21.702694 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.702696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.702757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.702833 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.703122 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.703656 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.704529 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.704751 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.704940 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.705828 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:21.705837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:21.705839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:21.705852 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:21.705872 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:21.705880 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:21.705966 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.706089 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.706150 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:21.709872 28407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:21.712982 28407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:21.728278 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35667 (local address 127.0.0.1:46192)
0929 00:55:21.722838 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.722907 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.723040 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.724026 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.724033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.724096 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:21.724283 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.724289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.725757 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.725761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.726827 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.726834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.727819 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.727822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.727922 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.727946 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.728014 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.728088 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":52,"spinlock_wait_cycles":28544,"thread_start_us":81,"threads_started":1}
I20260929 00:55:21.728495 28873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.11.40.193:35345)
0929 00:55:21.723292 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.723306 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.723332 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:21.723482 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.723488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.723680 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.723838 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:21.724342 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.725187 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.726685 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.727435 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.727549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.727774 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.728202 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.728321 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":101,"threads_started":1}
I20260929 00:55:21.728302 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:35345 (local address 127.0.0.1:50430)
0929 00:55:21.723247 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.723260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.723339 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.724026 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.724032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.724066 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:21.724209 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.724215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.725312 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.725315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.726575 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.726586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.727571 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.727574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.727784 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.727833 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.727944 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.728063 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":55,"spinlock_wait_cycles":3968,"thread_start_us":100,"threads_started":1}
I20260929 00:55:21.729209 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.11.40.194:35667)
0929 00:55:21.723600 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.723611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.723631 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:21.723701 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.723706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.723881 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.724024 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:21.724420 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.725625 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.726936 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.727681 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.728672 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.728906 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.728957 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.729058 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":166,"threads_started":1}
I20260929 00:55:21.730379 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41187 (local address 127.0.0.1:38076)
0929 00:55:21.722925 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.722943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.723040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.724026 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.724033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.724096 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:21.724274 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.724281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.726925 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.726927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.727782 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.727791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.729040 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.729043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.730082 (+ 1039us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.730106 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.730177 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.730247 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":149,"spinlock_wait_cycles":70400,"thread_start_us":126,"threads_started":1}
I20260929 00:55:21.731025 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.722709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.11.40.195:41187)
0929 00:55:21.723335 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.723349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.723375 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:21.723454 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.723459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.723683 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.723839 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:21.724412 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.725624 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.727905 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.728705 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.728829 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.728979 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.730762 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.730866 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":140,"threads_started":1}
I20260929 00:55:21.732030 28529 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 (DEFAULT_TABLE table=test-table [id=16d50eff76ce47b69b06447c335b5e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:21.732040 28660 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 (DEFAULT_TABLE table=test-table [id=16d50eff76ce47b69b06447c335b5e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:21.733067 28529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:21.733078 28660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:21.733848 28791 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 (DEFAULT_TABLE table=test-table [id=16d50eff76ce47b69b06447c335b5e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:21.734896 28791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:21.739665 28874 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Bootstrap starting.
I20260929 00:55:21.741585 28875 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Bootstrap starting.
I20260929 00:55:21.742148 28874 tablet_bootstrap.cc:654] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:21.742650 28876 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Bootstrap starting.
I20260929 00:55:21.743105 28874 log.cc:824] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:21.744217 28875 tablet_bootstrap.cc:654] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:21.744978 28876 tablet_bootstrap.cc:654] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:21.745184 28875 log.cc:824] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:21.745687 28874 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: No bootstrap required, opened a new log
I20260929 00:55:21.745849 28876 log.cc:824] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:21.745918 28874 ts_tablet_manager.cc:1397] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:55:21.747388 28875 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: No bootstrap required, opened a new log
I20260929 00:55:21.747602 28875 ts_tablet_manager.cc:1397] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:55:21.747917 28876 tablet_bootstrap.cc:492] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: No bootstrap required, opened a new log
I20260929 00:55:21.748121 28876 ts_tablet_manager.cc:1397] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 00:55:21.750524 28874 raft_consensus.cc:348] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.750813 28874 raft_consensus.cc:374] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:21.750900 28874 raft_consensus.cc:729] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5b09c8f8f8415ea3ac4998b90aa8ce, State: Initialized, Role: FOLLOWER
I20260929 00:55:21.751119 28875 raft_consensus.cc:348] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.751423 28874 consensus_queue.cc:260] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.751513 28875 raft_consensus.cc:374] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:21.751573 28875 raft_consensus.cc:729] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4e17dcf8fd44658f4892da23426dad, State: Initialized, Role: FOLLOWER
I20260929 00:55:21.752204 28875 consensus_queue.cc:260] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.752323 28876 raft_consensus.cc:348] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.752617 28876 raft_consensus.cc:374] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:21.752723 28876 raft_consensus.cc:729] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66cb44bcc45a4ac787b3f82d31159057, State: Initialized, Role: FOLLOWER
I20260929 00:55:21.752759 28874 ts_tablet_manager.cc:1428] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 00:55:21.753132 28595 heartbeater.cc:503] Master 127.11.40.254:44709 was elected leader, sending a full tablet report...
I20260929 00:55:21.753168 28876 consensus_queue.cc:260] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.753316 28875 ts_tablet_manager.cc:1428] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:21.753759 28857 heartbeater.cc:503] Master 127.11.40.254:44709 was elected leader, sending a full tablet report...
I20260929 00:55:21.754205 28726 heartbeater.cc:503] Master 127.11.40.254:44709 was elected leader, sending a full tablet report...
I20260929 00:55:21.755015 28876 ts_tablet_manager.cc:1428] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
W20260929 00:55:21.786278 28596 tablet.cc:2369] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:55:21.905494 28880 raft_consensus.cc:482] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:21.905813 28880 raft_consensus.cc:504] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.907011 28880 leader_election.cc:288] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d4e17dcf8fd44658f4892da23426dad (127.11.40.194:35667), 66cb44bcc45a4ac787b3f82d31159057 (127.11.40.195:41187)
I20260929 00:55:21.911670 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.907342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:36539 (local address 127.11.40.194:35667)
0929 00:55:21.907494 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.907502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.907689 (+ 187us) server_negotiation.cc:407] Connection header received
0929 00:55:21.907826 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.907831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.907909 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.908009 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:21.908415 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.909084 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.910449 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.910894 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.911032 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.911226 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.911449 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.911537 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:55:21.911811 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.907229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35667 (local address 127.11.40.193:36539)
0929 00:55:21.907600 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.907614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.907711 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.908045 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.908048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.908066 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:21.908211 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.908218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.909211 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.909215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.910192 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.910202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.911032 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.911036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.911293 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.911324 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.911410 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.911494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260929 00:55:21.911962 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.907493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:58645 (local address 127.11.40.195:41187)
0929 00:55:21.907626 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.907631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.907923 (+ 292us) server_negotiation.cc:407] Connection header received
0929 00:55:21.908063 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.908067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.908114 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.908204 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:21.908658 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.909232 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.910568 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.911095 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.911248 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.911302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.911745 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.911839 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:55:21.912341 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.907367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:41187 (local address 127.11.40.193:58645)
0929 00:55:21.907816 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.907830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.907928 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.908307 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.908310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.908364 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:21.908488 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.908494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.909396 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.909399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.910235 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.910390 (+ 155us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.911333 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.911337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.911474 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.911493 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:55:21.912103 (+ 610us) client_negotiation.cc:240] Negotiation successful
0929 00:55:21.912192 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":26,"thread_start_us":108,"threads_started":1}
I20260929 00:55:21.912603 28680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbc78b12f0c4192ba1cc13e2a2b96d8" candidate_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4e17dcf8fd44658f4892da23426dad" is_pre_election: true
I20260929 00:55:21.912995 28680 raft_consensus.cc:2454] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5b09c8f8f8415ea3ac4998b90aa8ce in term 0.
I20260929 00:55:21.913043 28811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbc78b12f0c4192ba1cc13e2a2b96d8" candidate_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cb44bcc45a4ac787b3f82d31159057" is_pre_election: true
I20260929 00:55:21.913369 28811 raft_consensus.cc:2454] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5b09c8f8f8415ea3ac4998b90aa8ce in term 0.
I20260929 00:55:21.913678 28484 leader_election.cc:302] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4e17dcf8fd44658f4892da23426dad, fb5b09c8f8f8415ea3ac4998b90aa8ce; no voters:
I20260929 00:55:21.914011 28880 raft_consensus.cc:2781] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:21.914095 28880 raft_consensus.cc:482] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:21.914126 28880 raft_consensus.cc:3037] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:21.915221 28880 raft_consensus.cc:504] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.915637 28880 leader_election.cc:288] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [CANDIDATE]: Term 1 election: Requested vote from peers 0d4e17dcf8fd44658f4892da23426dad (127.11.40.194:35667), 66cb44bcc45a4ac787b3f82d31159057 (127.11.40.195:41187)
I20260929 00:55:21.915999 28811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbc78b12f0c4192ba1cc13e2a2b96d8" candidate_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cb44bcc45a4ac787b3f82d31159057"
I20260929 00:55:21.916074 28680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbc78b12f0c4192ba1cc13e2a2b96d8" candidate_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4e17dcf8fd44658f4892da23426dad"
I20260929 00:55:21.916165 28811 raft_consensus.cc:3037] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:21.916231 28680 raft_consensus.cc:3037] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:21.917368 28811 raft_consensus.cc:2454] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5b09c8f8f8415ea3ac4998b90aa8ce in term 1.
I20260929 00:55:21.917538 28680 raft_consensus.cc:2454] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5b09c8f8f8415ea3ac4998b90aa8ce in term 1.
I20260929 00:55:21.917763 28485 leader_election.cc:302] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66cb44bcc45a4ac787b3f82d31159057, fb5b09c8f8f8415ea3ac4998b90aa8ce; no voters:
I20260929 00:55:21.918114 28880 raft_consensus.cc:2781] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:21.918375 28880 raft_consensus.cc:686] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [term 1 LEADER]: Becoming Leader. State: Replica: fb5b09c8f8f8415ea3ac4998b90aa8ce, State: Running, Role: LEADER
I20260929 00:55:21.918773 28880 consensus_queue.cc:237] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } }
I20260929 00:55:21.922148 28407 catalog_manager.cc:5495] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to fb5b09c8f8f8415ea3ac4998b90aa8ce (127.11.40.193). New cstate: current_term: 1 leader_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb5b09c8f8f8415ea3ac4998b90aa8ce" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 35345 } health_report { overall_health: HEALTHY } } }
W20260929 00:55:21.930405 28858 tablet.cc:2369] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:55:21.967218 28873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.962988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.11.40.193:35345)
0929 00:55:21.963139 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.963142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.963385 (+ 243us) server_negotiation.cc:407] Connection header received
0929 00:55:21.963494 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.963497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.963565 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.963676 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:21.963963 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.964538 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.965568 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.965812 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.965911 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.966303 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:21.966813 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:21.966923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:21.967041 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 00:55:21.967098 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:55:21.974050 28680 raft_consensus.cc:1261] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 1 FOLLOWER]: Refusing update from remote peer fb5b09c8f8f8415ea3ac4998b90aa8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:21.974050 28811 raft_consensus.cc:1261] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 1 FOLLOWER]: Refusing update from remote peer fb5b09c8f8f8415ea3ac4998b90aa8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:21.974933 28885 consensus_queue.cc:1035] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4e17dcf8fd44658f4892da23426dad" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:21.975238 28880 consensus_queue.cc:1035] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "66cb44bcc45a4ac787b3f82d31159057" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:21.984380 28890 mvcc.cc:205] Tried to move back new op lower bound from 7334475046796795904 to 7334475046582284288. Current Snapshot: MvccSnapshot[applied={T|T < 7334475046796795904}]
I20260929 00:55:21.997059 28405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:21.997282 28405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:22.000202 28405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4493d7bebef44d89fc2e7188dda6226: Sending DeleteTablet for 3 replicas of tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8
I20260929 00:55:22.001211 28660 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:21 UTC) from {username='slave'} at 127.0.0.1:46192
I20260929 00:55:22.001379 28791 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:21 UTC) from {username='slave'} at 127.0.0.1:38076
I20260929 00:55:22.001750 28898 tablet_replica.cc:330] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: stopping tablet replica
I20260929 00:55:22.001902 28899 tablet_replica.cc:330] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: stopping tablet replica
I20260929 00:55:22.001871 28529 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:21 UTC) from {username='slave'} at 127.0.0.1:50430
I20260929 00:55:22.002117 28898 raft_consensus.cc:2229] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:22.002208 28899 raft_consensus.cc:2229] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:22.002462 28898 raft_consensus.cc:2258] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:22.002508 28899 raft_consensus.cc:2258] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:22.003121 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28467
I20260929 00:55:22.003317 28900 tablet_replica.cc:330] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P fb5b09c8f8f8415ea3ac4998b90aa8ce: stopping tablet replica
I20260929 00:55:22.004247 28899 ts_tablet_manager.cc:1905] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:22.007247 28899 ts_tablet_manager.cc:1918] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:22.007383 28899 log.cc:1196] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/5bbc78b12f0c4192ba1cc13e2a2b96d8
I20260929 00:55:22.007764 28899 ts_tablet_manager.cc:1939] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 66cb44bcc45a4ac787b3f82d31159057: Deleting consensus metadata
I20260929 00:55:22.008664 28898 ts_tablet_manager.cc:1905] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:22.009259 28394 catalog_manager.cc:4841] TS 66cb44bcc45a4ac787b3f82d31159057 (127.11.40.195:41187): tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8 (table test-table [id=16d50eff76ce47b69b06447c335b5e67]) successfully deleted
I20260929 00:55:22.011094 28898 ts_tablet_manager.cc:1918] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:22.011269 28898 log.cc:1196] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/5bbc78b12f0c4192ba1cc13e2a2b96d8
W20260929 00:55:22.011353 28392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:35345 (error 108)
I20260929 00:55:22.011636 28898 ts_tablet_manager.cc:1939] T 5bbc78b12f0c4192ba1cc13e2a2b96d8 P 0d4e17dcf8fd44658f4892da23426dad: Deleting consensus metadata
I20260929 00:55:22.012338 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28598
I20260929 00:55:22.012593 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:22.012221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:35345 (local address 127.0.0.1:50442)
0929 00:55:22.012397 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:22.012486 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 00:55:22.013307 28392 catalog_manager.cc:4584] TS fb5b09c8f8f8415ea3ac4998b90aa8ce (127.11.40.193:35345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bbc78b12f0c4192ba1cc13e2a2b96d8: Network error: Client connection negotiation failed: client connection to 127.11.40.193:35345: connect: Connection refused (error 111)
I20260929 00:55:22.019165 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28729
I20260929 00:55:22.020057 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:22.019454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:35667 (local address 127.0.0.1:46202)
0929 00:55:22.019845 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:22.019914 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:35667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 00:55:22.025408 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28376
2026-09-29T00:55:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1212 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:22 dist-test-slave-5309 krb5kdc[28904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:22 dist-test-slave-5309 krb5kdc[28904](info): set up 2 sockets
Sep 29 00:55:22 dist-test-slave-5309 krb5kdc[28904](info): commencing operation
krb5kdc: starting...
W20260929 00:55:23.134966 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:23.273715 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44885
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:23.398330 28917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:23.398636 28917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:23.398680 28917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:23.402981 28917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:23.403064 28917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:23.403085 28917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:23.403103 28917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:23.403119 28917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:23.408757 28917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:44885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:23.409932 28917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:23.411047 28917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:23.419986 28925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:23.420017 28922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:23.419996 28923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:23.420426 28917 server_base.cc:1034] running on GCE node
I20260929 00:55:23.421049 28917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:23.422242 28917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:23.423439 28917 hybrid_clock.cc:648] HybridClock initialized: now 1790643323423408 us; error 43 us; skew 500 ppm
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:23.426776 28917 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:23.427903 28917 webserver.cc:460] Webserver started at http://127.11.40.254:33561/ using document root <none> and password file <none>
I20260929 00:55:23.428598 28917 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:23.428671 28917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:23.428972 28917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:23.431339 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "35651d9e736f4484bb7b81638628d04a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "31e59a96e192ec96021f6b55b82f04c2"
server_key_iv: "184d0d22e3eec5ea844e5d0db3fad894"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.431951 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "35651d9e736f4484bb7b81638628d04a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "31e59a96e192ec96021f6b55b82f04c2"
server_key_iv: "184d0d22e3eec5ea844e5d0db3fad894"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.436055 28917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:23.438844 28932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:23.440287 28917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:55:23.440543 28917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "35651d9e736f4484bb7b81638628d04a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "31e59a96e192ec96021f6b55b82f04c2"
server_key_iv: "184d0d22e3eec5ea844e5d0db3fad894"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.440680 28917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:23.453341 28917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:23.456418 28917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:23.456627 28917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:23.464346 28917 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:44885
I20260929 00:55:23.464373 28984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:44885 every 8 connection(s)
I20260929 00:55:23.465564 28917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:23.468609 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:23.472045 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 28917
I20260929 00:55:23.472214 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:23.472529 11427 external_mini_cluster.cc:1428] Setting key 1bcfb0bccbb8c6bc2835417f92052ee8
I20260929 00:55:23.475155 28985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: Bootstrap starting.
I20260929 00:55:23.477523 28985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:23.478299 28985 log.cc:824] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:23.480266 28985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: No bootstrap required, opened a new log
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:23.484558 28985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } }
I20260929 00:55:23.484822 28985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:23.484936 28985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35651d9e736f4484bb7b81638628d04a, State: Initialized, Role: FOLLOWER
I20260929 00:55:23.485426 28985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } }
I20260929 00:55:23.485580 28985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:23.485674 28985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:23.485772 28985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:23.486966 28985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } }
I20260929 00:55:23.487329 28985 leader_election.cc:302] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35651d9e736f4484bb7b81638628d04a; no voters:
I20260929 00:55:23.487794 28985 leader_election.cc:288] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:23.487911 28990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:23.488130 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.474173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.11.40.254:44885)
0929 00:55:23.474659 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:23.474683 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:23.474721 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:23.475476 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:23.475496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:23.475849 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:23.476191 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:23.477103 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.478166 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.479167 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.479647 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.479817 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.482698 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:23.482707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:23.482720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:23.482753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:23.484647 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:23.485231 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:23.485248 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:23.485252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:23.485363 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:23.485711 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:23.485715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:23.485716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:23.486201 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:23.486360 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:23.486730 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 00:55:23.487086 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260929 00:55:23.488155 28990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [term 1 LEADER]: Becoming Leader. State: Replica: 35651d9e736f4484bb7b81638628d04a, State: Running, Role: LEADER
I20260929 00:55:23.488473 28990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } }
I20260929 00:55:23.489334 28985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:23.489787 28992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35651d9e736f4484bb7b81638628d04a. Latest consensus state: current_term: 1 leader_uuid: "35651d9e736f4484bb7b81638628d04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } } }
I20260929 00:55:23.489903 28992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:23.490587 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35651d9e736f4484bb7b81638628d04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35651d9e736f4484bb7b81638628d04a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 44885 } } }
I20260929 00:55:23.490684 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:23.491989 28994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:23.494791 28994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:23.500941 28994 catalog_manager.cc:1347] Generated new cluster ID: 9f7a1cd1eeb74a0cba8a3de4bd5245d6
I20260929 00:55:23.501027 28994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:23.524012 28994 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:23.525488 28994 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:23.536900 28994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: Generated new TSK 0
I20260929 00:55:23.537686 28994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:55:23.582859 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:23.702927 29011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:23.703375 29011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:23.703451 29011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:23.708081 29011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:23.708221 29011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:23.708335 29011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:23.714154 29011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:23.715313 29011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:23.716373 29011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:23.726768 29016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:23.726768 29019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:23.727026 29011 server_base.cc:1034] running on GCE node
W20260929 00:55:23.726781 29017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:23.727608 29011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:23.728300 29011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:23.729558 29011 hybrid_clock.cc:648] HybridClock initialized: now 1790643323729523 us; error 45 us; skew 500 ppm
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:23.732800 29011 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:55:23.733824 29011 webserver.cc:460] Webserver started at http://127.11.40.193:38917/ using document root <none> and password file <none>
I20260929 00:55:23.734459 29011 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:23.734529 29011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:23.734777 29011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:23.737087 29011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "45bd50a0faaf4da79c0a8603d75589a1"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "894ab9e94a5359b6c620405fb441ffa4"
server_key_iv: "856498685b638b1e54e25ea76b38858d"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.737738 29011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "45bd50a0faaf4da79c0a8603d75589a1"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "894ab9e94a5359b6c620405fb441ffa4"
server_key_iv: "856498685b638b1e54e25ea76b38858d"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.742132 29011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:23.745040 29026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:23.746412 29011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:23.746562 29011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "45bd50a0faaf4da79c0a8603d75589a1"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "894ab9e94a5359b6c620405fb441ffa4"
server_key_iv: "856498685b638b1e54e25ea76b38858d"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.746693 29011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:23.763597 29011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:23.767127 29011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:23.767352 29011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:23.767999 29011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:23.768932 29011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:23.769004 29011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:23.769076 29011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:23.769121 29011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:23.778623 29011 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:40081
I20260929 00:55:23.778649 29139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:40081 every 8 connection(s)
I20260929 00:55:23.779992 29011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:23.780990 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29011
I20260929 00:55:23.781143 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:23.781490 11427 external_mini_cluster.cc:1428] Setting key a36093c36079739cec0a6a759e6bd58e
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:55:23.799674 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.782509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:43915 (local address 127.11.40.254:44885)
0929 00:55:23.782660 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:23.782664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:23.783930 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 00:55:23.785398 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:23.785401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:23.785461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:23.785543 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:23.787090 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.787683 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.789800 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.790399 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.790534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.794120 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:23.794125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:23.794127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:23.794164 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:23.796313 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:23.797290 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:23.797294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:23.797296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:23.797407 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:23.797916 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:23.797920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:23.797922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:23.798222 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:23.798342 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:23.799142 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 00:55:23.799332 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:23.800714 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.783106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44885 (local address 127.11.40.193:43915)
0929 00:55:23.783660 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:23.783723 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:23.785127 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:23.785736 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:23.785745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:23.786136 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:23.786865 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.786889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.787854 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.787860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.789609 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.789624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.790621 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.790627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.790871 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.791663 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:23.791685 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:23.793802 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:23.796517 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:23.796528 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:23.796549 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:23.797056 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:23.797566 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:23.797569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:23.797571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:23.797809 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:23.798367 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:23.798375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:23.798779 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 00:55:23.799041 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 00:55:23.799418 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":167,"threads_started":1}
I20260929 00:55:23.802134 29140 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44885
I20260929 00:55:23.802387 29140 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:23.802997 29140 heartbeater.cc:511] Master 127.11.40.254:44885 requested a full tablet report, sending...
I20260929 00:55:23.804708 28949 ts_manager.cc:194] Registered new tserver with Master: 45bd50a0faaf4da79c0a8603d75589a1 (127.11.40.193:40081)
I20260929 00:55:23.805974 28949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:43915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:55:23.816555 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:23.933408 29145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:23.933710 29145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:23.933794 29145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:23.937892 29145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:23.937995 29145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:23.938086 29145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:23.943665 29145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:23.944928 29145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:23.946089 29145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:23.956460 29150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:23.956586 29153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:23.956568 29151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:23.957460 29145 server_base.cc:1034] running on GCE node
I20260929 00:55:23.957978 29145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:23.958693 29145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:23.959916 29145 hybrid_clock.cc:648] HybridClock initialized: now 1790643323959884 us; error 46 us; skew 500 ppm
Sep 29 00:55:23 dist-test-slave-5309 krb5kdc[28904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:23.962993 29145 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:55:23.964021 29145 webserver.cc:460] Webserver started at http://127.11.40.194:35367/ using document root <none> and password file <none>
I20260929 00:55:23.964607 29145 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:23.964700 29145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:23.965004 29145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:23.967154 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "168c9b64c78e4586a04cd0c0b096ac0a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "1fb8a4daded50cad4476bf2bccc6cdc6"
server_key_iv: "24887eb0d55852bfc56b7c999037399c"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.967747 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "168c9b64c78e4586a04cd0c0b096ac0a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "1fb8a4daded50cad4476bf2bccc6cdc6"
server_key_iv: "24887eb0d55852bfc56b7c999037399c"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.971645 29145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:23.974418 29160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:23.975792 29145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:23.975960 29145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "168c9b64c78e4586a04cd0c0b096ac0a"
format_stamp: "Formatted at 2026-09-29 00:55:23 on dist-test-slave-5309"
server_key: "1fb8a4daded50cad4476bf2bccc6cdc6"
server_key_iv: "24887eb0d55852bfc56b7c999037399c"
server_key_version: "encryptionkey@0"
I20260929 00:55:23.976091 29145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:23.994987 29145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:23.998566 29145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:23.998790 29145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:23.999428 29145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:24.000281 29145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:24.000351 29145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.000420 29145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:24.000473 29145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.009891 29145 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:45719
I20260929 00:55:24.009912 29273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:45719 every 8 connection(s)
I20260929 00:55:24.010814 29145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:24.014001 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29145
I20260929 00:55:24.014132 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:24.014413 11427 external_mini_cluster.cc:1428] Setting key 35928ef0f4ff26876e5c9501e6ece7ec
Sep 29 00:55:24 dist-test-slave-5309 krb5kdc[28904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:55:24.025870 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.012772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:49869 (local address 127.11.40.254:44885)
0929 00:55:24.012939 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.012943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.013822 (+ 879us) server_negotiation.cc:407] Connection header received
0929 00:55:24.015418 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.015421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.015478 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.015562 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.016991 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.017531 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.018740 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.018985 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.019110 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.021946 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:24.021949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:24.021951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:24.021979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:24.023540 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.024119 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.024123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.024124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.024206 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.024580 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.024583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.024585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.024790 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:24.024959 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.025597 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.025741 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:55:24.026793 29276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.013198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44885 (local address 127.11.40.194:49869)
0929 00:55:24.013654 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.013698 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.015195 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.015738 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.015745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.016227 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.016738 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.016752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.017675 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.017679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.018598 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.018607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.019160 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.019164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.019356 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.020041 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:24.020058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.021719 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:24.023675 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:24.023682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:24.023700 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:24.023992 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:24.024308 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:24.024312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:24.024314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:24.024471 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:24.024967 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:24.024977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:24.025257 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.025516 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.025842 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:55:24.028183 29274 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44885
I20260929 00:55:24.028481 29274 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:24.029042 29274 heartbeater.cc:511] Master 127.11.40.254:44885 requested a full tablet report, sending...
I20260929 00:55:24.030185 28949 ts_manager.cc:194] Registered new tserver with Master: 168c9b64c78e4586a04cd0c0b096ac0a (127.11.40.194:45719)
I20260929 00:55:24.030969 28949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:49869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:55:24.048897 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:24.175539 29279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:24.175987 29279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:24.176093 29279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:24.182545 29279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:24.182644 29279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:24.182767 29279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:24.194278 29279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:44885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:24.195917 29279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:24.197502 29279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:24.209451 29285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:24.209452 29287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:24.209506 29279 server_base.cc:1034] running on GCE node
W20260929 00:55:24.209764 29284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:24.210254 29279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:24.210997 29279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:24.212193 29279 hybrid_clock.cc:648] HybridClock initialized: now 1790643324212154 us; error 51 us; skew 500 ppm
Sep 29 00:55:24 dist-test-slave-5309 krb5kdc[28904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643324, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:24.215551 29279 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:55:24.216661 29279 webserver.cc:460] Webserver started at http://127.11.40.195:43079/ using document root <none> and password file <none>
I20260929 00:55:24.217360 29279 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:24.217464 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:24.217715 29279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:24.219874 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "5164c4b8116e4e8292adfe85a6eb6ef0"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "8adc8b64dbb3336238122c814bfd7254"
server_key_iv: "bb16f603b0146fb3803cde55d070227c"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.220456 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "5164c4b8116e4e8292adfe85a6eb6ef0"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "8adc8b64dbb3336238122c814bfd7254"
server_key_iv: "bb16f603b0146fb3803cde55d070227c"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.224453 29279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:24.227078 29294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.228475 29279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:24.228615 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "5164c4b8116e4e8292adfe85a6eb6ef0"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "8adc8b64dbb3336238122c814bfd7254"
server_key_iv: "bb16f603b0146fb3803cde55d070227c"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.228736 29279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:24.242249 29279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:24.246114 29279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:24.246335 29279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:24.247026 29279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:24.247941 29279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:24.248030 29279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.248106 29279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:24.248158 29279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.257928 29279 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:34811
I20260929 00:55:24.257954 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:34811 every 8 connection(s)
I20260929 00:55:24.258876 29279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:24.266811 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29279
I20260929 00:55:24.267020 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:24.267328 11427 external_mini_cluster.cc:1428] Setting key a0f6a14ef1991948123806ab61d7587e
Sep 29 00:55:24 dist-test-slave-5309 krb5kdc[28904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643324, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:24.274013 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.260920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:54413 (local address 127.11.40.254:44885)
0929 00:55:24.261108 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.261112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.261930 (+ 818us) server_negotiation.cc:407] Connection header received
0929 00:55:24.263316 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.263319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.263378 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.263492 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.264743 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.265341 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.266518 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.266749 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.266888 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.270020 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:24.270023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:24.270025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:24.270050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:24.271490 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.272052 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.272055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.272057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.272194 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.272759 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.272762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.272763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.273061 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:24.273153 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.273736 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.273883 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:55:24.274964 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.261275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:44885 (local address 127.11.40.195:54413)
0929 00:55:24.261749 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.261794 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.263093 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.263675 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.263682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.264066 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.264592 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.264607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.265477 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.265481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.266382 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.266391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.266947 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.266951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.267170 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.267868 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:24.267903 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.269851 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:24.271612 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:24.271617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:24.271631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:24.271932 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:24.272321 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:24.272324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:24.272326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:24.272493 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:24.273186 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:24.273192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:24.273482 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.273659 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.273965 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260929 00:55:24.276265 29408 heartbeater.cc:347] Connected to a master server at 127.11.40.254:44885
I20260929 00:55:24.276499 29408 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:24.277154 29408 heartbeater.cc:511] Master 127.11.40.254:44885 requested a full tablet report, sending...
I20260929 00:55:24.278241 28949 ts_manager.cc:194] Registered new tserver with Master: 5164c4b8116e4e8292adfe85a6eb6ef0 (127.11.40.195:34811)
I20260929 00:55:24.278861 28949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:54413
I20260929 00:55:24.281844 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:24.293169 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.284361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.11.40.254:44885)
0929 00:55:24.284577 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.284582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.284740 (+ 158us) server_negotiation.cc:407] Connection header received
0929 00:55:24.284993 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.284996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.285043 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.285138 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.285900 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.286430 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.287300 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.287532 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.287646 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.289723 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:24.289728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:24.289734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:24.289789 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:24.291199 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.291764 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.291769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.291775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.291853 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:24.292192 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:24.292196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:24.292197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:24.292427 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:24.292594 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.292896 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.292995 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 00:55:24.297057 28949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:24.300273 28949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:24.317203 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.309844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:45719 (local address 127.0.0.1:51844)
0929 00:55:24.311106 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.311119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.311215 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.311737 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.311740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.311752 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:24.311864 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.311871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.314190 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.314195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.315174 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.315186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316482 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.316597 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.316642 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.316765 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.317037 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":97,"thread_start_us":107,"threads_started":1}
I20260929 00:55:24.317408 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.309551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:34811 (local address 127.0.0.1:41850)
0929 00:55:24.310544 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.310607 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.310767 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.311512 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.311518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.311543 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:24.311721 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.311731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.313323 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.313326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.315287 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.315300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316763 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.317008 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.317051 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.317136 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.317228 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":328,"thread_start_us":91,"threads_started":1}
I20260929 00:55:24.317961 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.309828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.11.40.193:40081)
0929 00:55:24.310263 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.310273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.311139 (+ 866us) server_negotiation.cc:407] Connection header received
0929 00:55:24.311250 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.311255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.311404 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.311524 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:24.311979 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.313151 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.315456 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316399 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.316687 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.317234 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.317795 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":92,"threads_started":1}
I20260929 00:55:24.317974 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.310415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.11.40.194:45719)
0929 00:55:24.310967 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.310985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.311314 (+ 329us) server_negotiation.cc:407] Connection header received
0929 00:55:24.311386 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.311390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.311594 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.311761 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:24.312974 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.314044 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.315390 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316342 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316481 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.317726 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.317773 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.317865 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":156,"threads_started":1}
I20260929 00:55:24.318045 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.310441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.11.40.195:34811)
0929 00:55:24.310914 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.310927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.310954 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:55:24.311033 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.311099 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.311326 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.311506 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:24.311874 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.313153 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.315453 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316399 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.316688 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.317770 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.317887 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":155,"threads_started":1}
I20260929 00:55:24.318548 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.309582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:40081 (local address 127.0.0.1:47350)
0929 00:55:24.311016 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.311032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.311135 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.311673 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.311675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.311720 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:24.311865 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.311871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.313322 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.313326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.315296 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.315303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.316763 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.316768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.317052 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.317073 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.318315 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.318407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"mutex_wait_us":468,"thread_start_us":96,"threads_started":1}
I20260929 00:55:24.320870 29342 tablet_service.cc:1467] Processing CreateTablet for tablet c31972ddaf2746b58558f4fe32eb1425 (DEFAULT_TABLE table=test-table [id=9d7eb03ce27940268324d5f08722d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:24.320981 29208 tablet_service.cc:1467] Processing CreateTablet for tablet c31972ddaf2746b58558f4fe32eb1425 (DEFAULT_TABLE table=test-table [id=9d7eb03ce27940268324d5f08722d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:24.321372 29074 tablet_service.cc:1467] Processing CreateTablet for tablet c31972ddaf2746b58558f4fe32eb1425 (DEFAULT_TABLE table=test-table [id=9d7eb03ce27940268324d5f08722d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:24.322036 29342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31972ddaf2746b58558f4fe32eb1425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:24.322045 29208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31972ddaf2746b58558f4fe32eb1425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:24.322268 29074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31972ddaf2746b58558f4fe32eb1425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:24.328305 29425 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Bootstrap starting.
I20260929 00:55:24.329962 29426 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: Bootstrap starting.
I20260929 00:55:24.331342 29425 tablet_bootstrap.cc:654] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:24.331722 29426 tablet_bootstrap.cc:654] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:24.332456 29426 log.cc:824] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:24.332508 29425 log.cc:824] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:24.333385 29427 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Bootstrap starting.
I20260929 00:55:24.334801 29426 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: No bootstrap required, opened a new log
I20260929 00:55:24.335008 29426 ts_tablet_manager.cc:1397] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 00:55:24.335673 29427 tablet_bootstrap.cc:654] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:24.337133 29427 log.cc:824] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:24.338229 29425 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: No bootstrap required, opened a new log
I20260929 00:55:24.338444 29425 ts_tablet_manager.cc:1397] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 00:55:24.339332 29427 tablet_bootstrap.cc:492] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: No bootstrap required, opened a new log
I20260929 00:55:24.339535 29427 ts_tablet_manager.cc:1397] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:55:24.339929 29426 raft_consensus.cc:348] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.340255 29426 raft_consensus.cc:374] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:24.340327 29426 raft_consensus.cc:729] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd50a0faaf4da79c0a8603d75589a1, State: Initialized, Role: FOLLOWER
I20260929 00:55:24.340899 29426 consensus_queue.cc:260] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.341413 29432 raft_consensus.cc:482] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:24.341603 29432 raft_consensus.cc:504] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.342031 29426 ts_tablet_manager.cc:1428] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260929 00:55:24.342438 29140 heartbeater.cc:503] Master 127.11.40.254:44885 was elected leader, sending a full tablet report...
I20260929 00:55:24.342779 29425 raft_consensus.cc:348] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.343066 29425 raft_consensus.cc:374] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:24.343122 29425 raft_consensus.cc:729] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5164c4b8116e4e8292adfe85a6eb6ef0, State: Initialized, Role: FOLLOWER
I20260929 00:55:24.343658 29425 consensus_queue.cc:260] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.343592 29427 raft_consensus.cc:348] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.343874 29427 raft_consensus.cc:374] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:24.343959 29427 raft_consensus.cc:729] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168c9b64c78e4586a04cd0c0b096ac0a, State: Initialized, Role: FOLLOWER
I20260929 00:55:24.344447 29427 consensus_queue.cc:260] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.344761 29425 ts_tablet_manager.cc:1428] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:24.345160 29408 heartbeater.cc:503] Master 127.11.40.254:44885 was elected leader, sending a full tablet report...
I20260929 00:55:24.345562 29427 ts_tablet_manager.cc:1428] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:55:24.346230 29274 heartbeater.cc:503] Master 127.11.40.254:44885 was elected leader, sending a full tablet report...
I20260929 00:55:24.346746 29432 leader_election.cc:288] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5164c4b8116e4e8292adfe85a6eb6ef0 (127.11.40.195:34811), 168c9b64c78e4586a04cd0c0b096ac0a (127.11.40.194:45719)
I20260929 00:55:24.352099 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.346547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:46237 (local address 127.11.40.194:45719)
0929 00:55:24.346811 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.346829 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.347049 (+ 220us) server_negotiation.cc:407] Connection header received
0929 00:55:24.347263 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.347266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.347322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.347374 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:24.349156 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.349725 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.351030 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.351548 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.351681 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.351734 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.351905 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.351975 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260929 00:55:24.352123 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.346486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:45719 (local address 127.11.40.193:46237)
0929 00:55:24.346966 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.346980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.347130 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.348398 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.348400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.348419 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:24.349023 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.349030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.349887 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.349890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.350911 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.350921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.351670 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.351674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.351772 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.351800 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.351890 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.351962 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":113,"threads_started":1}
I20260929 00:55:24.352716 29228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31972ddaf2746b58558f4fe32eb1425" candidate_uuid: "45bd50a0faaf4da79c0a8603d75589a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" is_pre_election: true
I20260929 00:55:24.353010 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.347214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:36885 (local address 127.11.40.195:34811)
0929 00:55:24.347547 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.347551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.347564 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:55:24.347611 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.347613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.347655 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.347730 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:24.349799 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.350532 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.351795 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.352254 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.352384 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.352434 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.352777 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.352882 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260929 00:55:24.353036 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.346458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:34811 (local address 127.11.40.193:36885)
0929 00:55:24.346776 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.346794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.347000 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.348614 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.348618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.348643 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:24.348785 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.348793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.350747 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.350751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.351619 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.351629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.352545 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.352548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.352627 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.352649 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.352761 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.352879 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
I20260929 00:55:24.353044 29228 raft_consensus.cc:2454] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd50a0faaf4da79c0a8603d75589a1 in term 0.
I20260929 00:55:24.353660 29029 leader_election.cc:302] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 168c9b64c78e4586a04cd0c0b096ac0a, 45bd50a0faaf4da79c0a8603d75589a1; no voters:
I20260929 00:55:24.353888 29362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31972ddaf2746b58558f4fe32eb1425" candidate_uuid: "45bd50a0faaf4da79c0a8603d75589a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" is_pre_election: true
I20260929 00:55:24.353991 29431 raft_consensus.cc:2781] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:24.354112 29431 raft_consensus.cc:482] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:24.354172 29431 raft_consensus.cc:3037] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:24.354198 29362 raft_consensus.cc:2454] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd50a0faaf4da79c0a8603d75589a1 in term 0.
I20260929 00:55:24.355368 29431 raft_consensus.cc:504] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.355764 29431 leader_election.cc:288] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [CANDIDATE]: Term 1 election: Requested vote from peers 5164c4b8116e4e8292adfe85a6eb6ef0 (127.11.40.195:34811), 168c9b64c78e4586a04cd0c0b096ac0a (127.11.40.194:45719)
I20260929 00:55:24.356081 29362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31972ddaf2746b58558f4fe32eb1425" candidate_uuid: "45bd50a0faaf4da79c0a8603d75589a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0"
I20260929 00:55:24.356212 29362 raft_consensus.cc:3037] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:24.356228 29228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31972ddaf2746b58558f4fe32eb1425" candidate_uuid: "45bd50a0faaf4da79c0a8603d75589a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "168c9b64c78e4586a04cd0c0b096ac0a"
I20260929 00:55:24.356400 29228 raft_consensus.cc:3037] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:24.357628 29362 raft_consensus.cc:2454] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd50a0faaf4da79c0a8603d75589a1 in term 1.
I20260929 00:55:24.357729 29228 raft_consensus.cc:2454] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd50a0faaf4da79c0a8603d75589a1 in term 1.
I20260929 00:55:24.358032 29029 leader_election.cc:302] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 168c9b64c78e4586a04cd0c0b096ac0a, 45bd50a0faaf4da79c0a8603d75589a1; no voters:
I20260929 00:55:24.358217 29431 raft_consensus.cc:2781] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:24.358335 29431 raft_consensus.cc:686] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 1 LEADER]: Becoming Leader. State: Replica: 45bd50a0faaf4da79c0a8603d75589a1, State: Running, Role: LEADER
I20260929 00:55:24.358654 29431 consensus_queue.cc:237] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } }
I20260929 00:55:24.362099 28949 catalog_manager.cc:5495] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45bd50a0faaf4da79c0a8603d75589a1 (127.11.40.193). New cstate: current_term: 1 leader_uuid: "45bd50a0faaf4da79c0a8603d75589a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45bd50a0faaf4da79c0a8603d75589a1" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 40081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:24.376385 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.372368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.11.40.193:40081)
0929 00:55:24.372540 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.372545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.372568 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:55:24.372693 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.372696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.372761 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.372929 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:24.373346 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.373889 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.374755 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.375001 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.375122 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.375458 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:24.375896 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:24.376068 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.376178 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.376224 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:55:24.383504 29228 raft_consensus.cc:1261] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 1 FOLLOWER]: Refusing update from remote peer 45bd50a0faaf4da79c0a8603d75589a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:24.383500 29362 raft_consensus.cc:1261] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 1 FOLLOWER]: Refusing update from remote peer 45bd50a0faaf4da79c0a8603d75589a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:24.384349 29432 consensus_queue.cc:1035] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5164c4b8116e4e8292adfe85a6eb6ef0" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:24.384569 29431 consensus_queue.cc:1035] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "168c9b64c78e4586a04cd0c0b096ac0a" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:24.392089 29440 mvcc.cc:205] Tried to move back new op lower bound from 7334475056666849280 to 7334475056576217088. Current Snapshot: MvccSnapshot[applied={T|T < 7334475056666849280}]
I20260929 00:55:24.392665 29439 mvcc.cc:205] Tried to move back new op lower bound from 7334475056666849280 to 7334475056576217088. Current Snapshot: MvccSnapshot[applied={T|T < 7334475056666849280}]
I20260929 00:55:24.402843 28949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:24.403048 28949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:24.405740 28949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35651d9e736f4484bb7b81638628d04a: Sending DeleteTablet for 3 replicas of tablet c31972ddaf2746b58558f4fe32eb1425
I20260929 00:55:24.406625 29342 tablet_service.cc:1514] Processing DeleteTablet for tablet c31972ddaf2746b58558f4fe32eb1425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:24 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:41850
I20260929 00:55:24.406621 29074 tablet_service.cc:1514] Processing DeleteTablet for tablet c31972ddaf2746b58558f4fe32eb1425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:24 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:47350
I20260929 00:55:24.407514 29450 tablet_replica.cc:330] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1: stopping tablet replica
I20260929 00:55:24.407501 29208 tablet_service.cc:1514] Processing DeleteTablet for tablet c31972ddaf2746b58558f4fe32eb1425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:24 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:51844
I20260929 00:55:24.407622 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29011
I20260929 00:55:24.407876 29450 raft_consensus.cc:2229] T c31972ddaf2746b58558f4fe32eb1425 P 45bd50a0faaf4da79c0a8603d75589a1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:24.408376 29449 tablet_replica.cc:330] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: stopping tablet replica
I20260929 00:55:24.408602 29449 raft_consensus.cc:2229] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:24.408929 29449 raft_consensus.cc:2258] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:24.410315 29449 ts_tablet_manager.cc:1905] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:24.410846 29451 tablet_replica.cc:330] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: stopping tablet replica
I20260929 00:55:24.411159 29451 raft_consensus.cc:2229] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:24.411481 29451 raft_consensus.cc:2258] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:24.412770 29449 ts_tablet_manager.cc:1918] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:24.412943 29449 log.cc:1196] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/c31972ddaf2746b58558f4fe32eb1425
I20260929 00:55:24.413089 29451 ts_tablet_manager.cc:1905] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:24.413328 29449 ts_tablet_manager.cc:1939] T c31972ddaf2746b58558f4fe32eb1425 P 5164c4b8116e4e8292adfe85a6eb6ef0: Deleting consensus metadata
I20260929 00:55:24.414450 28936 catalog_manager.cc:4841] TS 5164c4b8116e4e8292adfe85a6eb6ef0 (127.11.40.195:34811): tablet c31972ddaf2746b58558f4fe32eb1425 (table test-table [id=9d7eb03ce27940268324d5f08722d7e2]) successfully deleted
I20260929 00:55:24.416551 29451 ts_tablet_manager.cc:1918] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:24.416661 29451 log.cc:1196] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/c31972ddaf2746b58558f4fe32eb1425
W20260929 00:55:24.416846 28933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:40081 (error 108)
I20260929 00:55:24.417039 29451 ts_tablet_manager.cc:1939] T c31972ddaf2746b58558f4fe32eb1425 P 168c9b64c78e4586a04cd0c0b096ac0a: Deleting consensus metadata
I20260929 00:55:24.417908 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29145
I20260929 00:55:24.418169 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.417717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:40081 (local address 127.0.0.1:47376)
0929 00:55:24.417972 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.418058 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260929 00:55:24.419754 28933 catalog_manager.cc:4584] TS 45bd50a0faaf4da79c0a8603d75589a1 (127.11.40.193:40081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c31972ddaf2746b58558f4fe32eb1425: Network error: Client connection negotiation failed: client connection to 127.11.40.193:40081: connect: Connection refused (error 111)
I20260929 00:55:24.424556 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29279
I20260929 00:55:24.426635 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.424760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:45719 (local address 127.0.0.1:51848)
0929 00:55:24.426433 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.426512 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:45719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1541}
I20260929 00:55:24.430958 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 28917
2026-09-29T00:55:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2405 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T00:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:24Z Disabled control of system clock
I20260929 00:55:24.462759 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33073
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:24.579495 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:24.579810 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:24.579849 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:24.584281 29456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:24.584371 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:24.584388 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:24.584405 29456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:24.584421 29456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:24.589510 29456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:24.590770 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:24.592204 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:24.601787 29461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:24.601943 29456 server_base.cc:1034] running on GCE node
W20260929 00:55:24.601856 29462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:24.602152 29464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:24.602869 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:24.604061 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:24.605263 29456 hybrid_clock.cc:648] HybridClock initialized: now 1790643324605251 us; error 28 us; skew 500 ppm
I20260929 00:55:24.607404 29456 webserver.cc:460] Webserver started at http://127.11.40.254:41345/ using document root <none> and password file <none>
I20260929 00:55:24.608048 29456 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:24.608150 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:24.608398 29456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:24.610661 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "d3adc4171b274af3bdc5fc332588a2a7"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "c8b0c9c6784ac40cc7954685147d69c6"
server_key_iv: "67b4e8f3dac6369b7f78050a2f5cf7eb"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.611312 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "d3adc4171b274af3bdc5fc332588a2a7"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "c8b0c9c6784ac40cc7954685147d69c6"
server_key_iv: "67b4e8f3dac6369b7f78050a2f5cf7eb"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.615528 29456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 00:55:24.618352 29470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.619805 29456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:24.619992 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "d3adc4171b274af3bdc5fc332588a2a7"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "c8b0c9c6784ac40cc7954685147d69c6"
server_key_iv: "67b4e8f3dac6369b7f78050a2f5cf7eb"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.620137 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:24.631788 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:24.632516 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:24.632795 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:24.640196 29456 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33073
I20260929 00:55:24.640221 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33073 every 8 connection(s)
I20260929 00:55:24.641386 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:24.644690 29523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:24.650568 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: Bootstrap starting.
I20260929 00:55:24.651508 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29456
I20260929 00:55:24.651660 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:24.651947 11427 external_mini_cluster.cc:1428] Setting key e29ae3ec5260ee26edbf6caf3e5743ec
I20260929 00:55:24.653514 29523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:24.654466 29523 log.cc:824] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:24.656939 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: No bootstrap required, opened a new log
I20260929 00:55:24.661628 29523 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } }
I20260929 00:55:24.661950 29523 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:24.662007 29523 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adc4171b274af3bdc5fc332588a2a7, State: Initialized, Role: FOLLOWER
I20260929 00:55:24.662529 29523 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } }
I20260929 00:55:24.662708 29523 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:24.662789 29523 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:24.662899 29523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:24.663012 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.653179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.11.40.254:33073)
0929 00:55:24.653974 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.653995 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.654043 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:55:24.654992 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.655013 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.655409 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.655812 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.656271 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.657769 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.658784 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.659507 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.659696 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.660566 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:24.660576 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:24.660601 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:24.661201 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:24.661294 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:24.661323 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:24.661438 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.661623 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.661929 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":127,"threads_started":1}
I20260929 00:55:24.664875 29523 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } }
I20260929 00:55:24.665277 29523 leader_election.cc:302] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3adc4171b274af3bdc5fc332588a2a7; no voters:
I20260929 00:55:24.665659 29523 leader_election.cc:288] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:24.665797 29528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:24.666028 29528 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [term 1 LEADER]: Becoming Leader. State: Replica: d3adc4171b274af3bdc5fc332588a2a7, State: Running, Role: LEADER
I20260929 00:55:24.666411 29528 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } }
I20260929 00:55:24.667039 29523 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:24.668144 29530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3adc4171b274af3bdc5fc332588a2a7. Latest consensus state: current_term: 1 leader_uuid: "d3adc4171b274af3bdc5fc332588a2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } } }
I20260929 00:55:24.668282 29530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:24.668624 29529 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3adc4171b274af3bdc5fc332588a2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adc4171b274af3bdc5fc332588a2a7" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33073 } } }
I20260929 00:55:24.668706 29529 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:24.668751 29537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:24.671967 29537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:24.677917 29537 catalog_manager.cc:1347] Generated new cluster ID: 4c4cee3e23f1418c9b15c82fc962246e
I20260929 00:55:24.678002 29537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:24.698040 29537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:24.699083 29537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:24.705703 29537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: Generated new TSK 0
I20260929 00:55:24.706574 29537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:24.714326 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--builtin_ntp_servers=127.11.40.212:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:24.831511 29547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:24.831836 29547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:24.831909 29547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:24.836077 29547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:24.836186 29547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:24.836289 29547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:24.841349 29547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:24.842550 29547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:24.844003 29547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:24.854411 29552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:24.854358 29555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:24.854358 29553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:24.854540 29547 server_base.cc:1034] running on GCE node
I20260929 00:55:24.855094 29547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:24.855829 29547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:24.857048 29547 hybrid_clock.cc:648] HybridClock initialized: now 1790643324857012 us; error 49 us; skew 500 ppm
I20260929 00:55:24.872193 29547 webserver.cc:460] Webserver started at http://127.11.40.193:45479/ using document root <none> and password file <none>
I20260929 00:55:24.872956 29547 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:24.873064 29547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:24.873355 29547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:24.875602 29547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "9f94b579975744b18492da0349b9949f"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "88e3ba614ba879047a83741a0e80210c"
server_key_iv: "2d038feb52cbb908aea75958b9b918df"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.876196 29547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "9f94b579975744b18492da0349b9949f"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "88e3ba614ba879047a83741a0e80210c"
server_key_iv: "2d038feb52cbb908aea75958b9b918df"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.880574 29547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:24.883615 29561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.885377 29547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:24.885535 29547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "9f94b579975744b18492da0349b9949f"
format_stamp: "Formatted at 2026-09-29 00:55:24 on dist-test-slave-5309"
server_key: "88e3ba614ba879047a83741a0e80210c"
server_key_iv: "2d038feb52cbb908aea75958b9b918df"
server_key_version: "encryptionkey@0"
I20260929 00:55:24.885673 29547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:24.903798 29547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:24.904585 29547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:24.904798 29547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:24.905485 29547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:24.906457 29547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:24.906527 29547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.906596 29547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:24.906644 29547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:24.917655 29547 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:42669
I20260929 00:55:24.917686 29674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:42669 every 8 connection(s)
I20260929 00:55:24.918850 29547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:24.921836 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29547
I20260929 00:55:24.921988 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:24.922336 11427 external_mini_cluster.cc:1428] Setting key a2c9904b6182532e50a95e3024aa0b26
I20260929 00:55:24.924754 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--builtin_ntp_servers=127.11.40.212:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:24.930203 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.920790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:55009 (local address 127.11.40.254:33073)
0929 00:55:24.920994 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.920999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.922375 (+ 1376us) server_negotiation.cc:407] Connection header received
0929 00:55:24.924100 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.924104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.924185 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.924261 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.925429 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.926045 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.927791 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.928042 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.928137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.929633 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:24.929636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:24.929639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:24.929654 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:24.929674 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:24.929683 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:24.929784 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:24.930023 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 00:55:24.930083 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 00:55:24.931546 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.921269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33073 (local address 127.11.40.193:55009)
0929 00:55:24.922161 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.922226 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.923852 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.924467 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.924475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.924580 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.925235 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.925252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.926197 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.926200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.927619 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.927636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.928219 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.928226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.928471 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.929267 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:24.929272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.929338 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:24.929442 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:24.929792 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:24.929803 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:24.929812 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:55:24.929964 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 00:55:24.930327 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":122,"threads_started":1}
I20260929 00:55:24.933101 29675 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33073
I20260929 00:55:24.933439 29675 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:24.934067 29675 heartbeater.cc:511] Master 127.11.40.254:33073 requested a full tablet report, sending...
I20260929 00:55:24.936153 29484 ts_manager.cc:194] Registered new tserver with Master: 9f94b579975744b18492da0349b9949f (127.11.40.193:42669)
I20260929 00:55:24.937474 29484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.193:55009
W20260929 00:55:25.050050 29678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:25.050354 29678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:25.050400 29678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:25.055135 29678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:25.055202 29678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:25.055264 29678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:25.060139 29678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:25.061342 29678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:25.062763 29678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:25.072947 29686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:25.072940 29684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:25.072935 29683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:25.073230 29678 server_base.cc:1034] running on GCE node
I20260929 00:55:25.073837 29678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:25.074733 29678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:25.075950 29678 hybrid_clock.cc:648] HybridClock initialized: now 1790643325075919 us; error 52 us; skew 500 ppm
I20260929 00:55:25.078235 29678 webserver.cc:460] Webserver started at http://127.11.40.194:37035/ using document root <none> and password file <none>
I20260929 00:55:25.078913 29678 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:25.078987 29678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:25.079231 29678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:25.081583 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "f54db8d94e0048d4b8106689f92651fe"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "0c871ed43fd7b4546663f209083f4852"
server_key_iv: "4c776bf4e11d9d1f1bc58ebaa2c3570f"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.082207 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "f54db8d94e0048d4b8106689f92651fe"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "0c871ed43fd7b4546663f209083f4852"
server_key_iv: "4c776bf4e11d9d1f1bc58ebaa2c3570f"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.086457 29678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:25.089514 29692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.090981 29678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:25.091126 29678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "f54db8d94e0048d4b8106689f92651fe"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "0c871ed43fd7b4546663f209083f4852"
server_key_iv: "4c776bf4e11d9d1f1bc58ebaa2c3570f"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.091271 29678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:25.105441 29678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:25.106202 29678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:25.106420 29678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:25.107038 29678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:25.108039 29678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:25.108111 29678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.108184 29678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:25.108235 29678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.117600 29678 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:39583
I20260929 00:55:25.117650 29806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:39583 every 8 connection(s)
I20260929 00:55:25.118620 29678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:25.123078 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29678
I20260929 00:55:25.123250 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:25.123603 11427 external_mini_cluster.cc:1428] Setting key 26ad34fe15fd9e7e4c49d82322156278
I20260929 00:55:25.126480 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--builtin_ntp_servers=127.11.40.212:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:25.130336 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.120763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:34523 (local address 127.11.40.254:33073)
0929 00:55:25.120994 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.120998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.122053 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 00:55:25.123898 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.123902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.123960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.124118 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.125307 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.126022 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.128098 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.128318 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.128409 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.129772 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:25.129775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:25.129778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:25.129792 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:25.129820 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:25.129829 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:25.129922 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.130105 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.130177 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 00:55:25.131470 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.120998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33073 (local address 127.11.40.194:34523)
0929 00:55:25.121872 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.121917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.123583 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.124272 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.124279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.124380 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:25.125144 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.125171 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.126205 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.126213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.127930 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.127967 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.128485 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.128488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.128659 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.129456 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:25.129458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:25.129506 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:25.129582 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:25.129924 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:25.129937 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:25.129946 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.130092 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.130436 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":126,"threads_started":1}
I20260929 00:55:25.133594 29807 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33073
I20260929 00:55:25.133912 29807 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:25.134517 29807 heartbeater.cc:511] Master 127.11.40.254:33073 requested a full tablet report, sending...
I20260929 00:55:25.135699 29484 ts_manager.cc:194] Registered new tserver with Master: f54db8d94e0048d4b8106689f92651fe (127.11.40.194:39583)
I20260929 00:55:25.136530 29484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.194:34523
W20260929 00:55:25.247150 29810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:25.247499 29810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:25.247577 29810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:25.251747 29810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:25.251849 29810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:25.251945 29810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:25.256958 29810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:25.258149 29810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:25.259652 29810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:25.269697 29818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:25.269693 29816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:25.269707 29815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:25.270143 29810 server_base.cc:1034] running on GCE node
I20260929 00:55:25.270608 29810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:25.271291 29810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:25.272518 29810 hybrid_clock.cc:648] HybridClock initialized: now 1790643325272493 us; error 39 us; skew 500 ppm
I20260929 00:55:25.274682 29810 webserver.cc:460] Webserver started at http://127.11.40.195:35135/ using document root <none> and password file <none>
I20260929 00:55:25.275302 29810 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:25.275399 29810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:25.275640 29810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:25.278265 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "18da0337540943738f038e1bf4d24ac8"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "5bcd42a499130e70c9289704dedb5488"
server_key_iv: "f8db7eb18cc5839eb72273f739bda2d8"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.278930 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "18da0337540943738f038e1bf4d24ac8"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "5bcd42a499130e70c9289704dedb5488"
server_key_iv: "f8db7eb18cc5839eb72273f739bda2d8"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.283015 29810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:25.285715 29825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.287109 29810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:25.287257 29810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "18da0337540943738f038e1bf4d24ac8"
format_stamp: "Formatted at 2026-09-29 00:55:25 on dist-test-slave-5309"
server_key: "5bcd42a499130e70c9289704dedb5488"
server_key_iv: "f8db7eb18cc5839eb72273f739bda2d8"
server_key_version: "encryptionkey@0"
I20260929 00:55:25.287388 29810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:25.302276 29810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:25.303015 29810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:25.303217 29810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:25.303815 29810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:25.304788 29810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:25.304922 29810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.304993 29810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:25.305038 29810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:25.314003 29810 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:33625
I20260929 00:55:25.314033 29938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:33625 every 8 connection(s)
I20260929 00:55:25.314918 29810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:25.324610 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.316843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:59177 (local address 127.11.40.254:33073)
0929 00:55:25.316994 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.316998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.318048 (+ 1050us) server_negotiation.cc:407] Connection header received
0929 00:55:25.319499 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.319503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.319564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.319673 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.320598 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.321247 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.322471 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.322706 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.322829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.323965 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:25.323968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:25.323970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:25.324004 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:25.324038 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:25.324053 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:25.324145 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.324368 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.324459 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:55:25.324651 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 29810
I20260929 00:55:25.324783 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:25.325124 11427 external_mini_cluster.cc:1428] Setting key 71e7688eb339245ae302bd2ef4f17ea2
I20260929 00:55:25.325752 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.317207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33073 (local address 127.11.40.195:59177)
0929 00:55:25.317887 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.317929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.319269 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.319854 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.319861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.319949 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:25.320442 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.320455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.321399 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.321402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.322337 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.322346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.322873 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.322877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.323051 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.323649 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:25.323651 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:25.323693 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:25.323765 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:25.324147 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:25.324159 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:25.324167 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.324376 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.324674 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260929 00:55:25.326928 29939 heartbeater.cc:347] Connected to a master server at 127.11.40.254:33073
I20260929 00:55:25.327167 29939 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:25.327708 29939 heartbeater.cc:511] Master 127.11.40.254:33073 requested a full tablet report, sending...
I20260929 00:55:25.328850 29484 ts_manager.cc:194] Registered new tserver with Master: 18da0337540943738f038e1bf4d24ac8 (127.11.40.195:33625)
I20260929 00:55:25.329485 29484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.195:59177
I20260929 00:55:25.339241 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:25.345268 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.341342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.11.40.254:33073)
0929 00:55:25.341492 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.341497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.341690 (+ 193us) server_negotiation.cc:407] Connection header received
0929 00:55:25.341785 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.341787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.341828 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.341897 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.342167 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.342690 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.343497 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.343709 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.343813 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.344653 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:25.344656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:25.344658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:25.344669 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:25.344689 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:25.344697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:25.344897 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.345021 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.345085 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:55:25.348429 29484 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:25.351501 29484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:25.367170 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.361881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.11.40.195:33625)
0929 00:55:25.362248 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.362260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.362530 (+ 270us) server_negotiation.cc:407] Connection header received
0929 00:55:25.362616 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.362621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.362827 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.362992 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:25.363385 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.364515 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.365718 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.366426 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.366542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.366742 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.366871 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.366991 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260929 00:55:25.367182 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.361111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:33625 (local address 127.0.0.1:36746)
0929 00:55:25.362401 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.362414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.362508 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.363133 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.363137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.363155 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:25.363250 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.363256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.364643 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.364646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.365589 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.365598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.366555 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.366558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.366666 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.366725 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.366818 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.367025 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":92,"threads_started":1}
I20260929 00:55:25.367587 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.360840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.11.40.194:39583)
0929 00:55:25.361379 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.361392 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.361859 (+ 467us) server_negotiation.cc:407] Connection header received
0929 00:55:25.362096 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.362102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.362275 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.362383 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:25.362932 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.363920 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.366079 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.366891 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.366989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.367157 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.367310 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.367427 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":179,"threads_started":1}
I20260929 00:55:25.368184 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.360635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39583 (local address 127.0.0.1:59676)
0929 00:55:25.361701 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.361760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.361922 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.362598 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.362604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.362630 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:25.362775 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.362782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.364046 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.364049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.365944 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.365949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.367019 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.367023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.367120 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.367154 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.367826 (+ 672us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.368026 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":72,"thread_start_us":103,"threads_started":1}
I20260929 00:55:25.368844 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.360770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:42669 (local address 127.0.0.1:56752)
0929 00:55:25.361701 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.361760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.361901 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.362775 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.362777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.362795 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:25.362899 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.362908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.364246 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.364250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.365937 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.365949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.368176 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.368180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.368497 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.368516 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.368621 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.368694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":297,"spinlock_wait_cycles":3840,"thread_start_us":98,"threads_started":1}
I20260929 00:55:25.368957 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.361490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.11.40.193:42669)
0929 00:55:25.362320 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.362330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.362352 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:55:25.362432 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.362440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.362643 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.362808 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:25.363026 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.364115 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.367224 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.368050 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.368177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.368352 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.368659 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.368760 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":136,"threads_started":1}
I20260929 00:55:25.371096 29873 tablet_service.cc:1467] Processing CreateTablet for tablet e02c0cb93710428aac10b07a2cb75573 (DEFAULT_TABLE table=test-table [id=3fb94148564f49a8b7174e2cb79adf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:25.371490 29741 tablet_service.cc:1467] Processing CreateTablet for tablet e02c0cb93710428aac10b07a2cb75573 (DEFAULT_TABLE table=test-table [id=3fb94148564f49a8b7174e2cb79adf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:25.372175 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c0cb93710428aac10b07a2cb75573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:25.372537 29741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c0cb93710428aac10b07a2cb75573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:25.372202 29609 tablet_service.cc:1467] Processing CreateTablet for tablet e02c0cb93710428aac10b07a2cb75573 (DEFAULT_TABLE table=test-table [id=3fb94148564f49a8b7174e2cb79adf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:25.373236 29609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c0cb93710428aac10b07a2cb75573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:25.378557 29957 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Bootstrap starting.
I20260929 00:55:25.380327 29959 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Bootstrap starting.
I20260929 00:55:25.381165 29958 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Bootstrap starting.
I20260929 00:55:25.381691 29957 tablet_bootstrap.cc:654] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:25.382421 29957 log.cc:824] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:25.382550 29959 tablet_bootstrap.cc:654] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:25.383229 29958 tablet_bootstrap.cc:654] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:25.383364 29959 log.cc:824] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:25.383989 29958 log.cc:824] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:25.384855 29957 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: No bootstrap required, opened a new log
I20260929 00:55:25.385061 29957 ts_tablet_manager.cc:1397] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:55:25.385493 29959 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: No bootstrap required, opened a new log
I20260929 00:55:25.385716 29959 ts_tablet_manager.cc:1397] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:55:25.386734 29958 tablet_bootstrap.cc:492] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: No bootstrap required, opened a new log
I20260929 00:55:25.386930 29958 ts_tablet_manager.cc:1397] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:55:25.389169 29959 raft_consensus.cc:348] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.389483 29959 raft_consensus.cc:374] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:25.389542 29959 raft_consensus.cc:729] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f94b579975744b18492da0349b9949f, State: Initialized, Role: FOLLOWER
I20260929 00:55:25.389976 29959 consensus_queue.cc:260] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.390089 29957 raft_consensus.cc:348] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.390354 29957 raft_consensus.cc:374] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:25.390419 29957 raft_consensus.cc:729] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18da0337540943738f038e1bf4d24ac8, State: Initialized, Role: FOLLOWER
I20260929 00:55:25.390851 29675 heartbeater.cc:503] Master 127.11.40.254:33073 was elected leader, sending a full tablet report...
I20260929 00:55:25.391078 29959 ts_tablet_manager.cc:1428] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 00:55:25.391453 29958 raft_consensus.cc:348] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.391703 29958 raft_consensus.cc:374] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:25.391770 29958 raft_consensus.cc:729] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54db8d94e0048d4b8106689f92651fe, State: Initialized, Role: FOLLOWER
I20260929 00:55:25.392189 29958 consensus_queue.cc:260] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.393225 29958 ts_tablet_manager.cc:1428] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 00:55:25.394166 29807 heartbeater.cc:503] Master 127.11.40.254:33073 was elected leader, sending a full tablet report...
I20260929 00:55:25.395702 29957 consensus_queue.cc:260] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.396502 29939 heartbeater.cc:503] Master 127.11.40.254:33073 was elected leader, sending a full tablet report...
I20260929 00:55:25.397030 29957 ts_tablet_manager.cc:1428] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
W20260929 00:55:25.420078 29676 tablet.cc:2369] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:55:25.471397 29963 raft_consensus.cc:482] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:25.471674 29963 raft_consensus.cc:504] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.472975 29963 leader_election.cc:288] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54db8d94e0048d4b8106689f92651fe (127.11.40.194:39583), 18da0337540943738f038e1bf4d24ac8 (127.11.40.195:33625)
I20260929 00:55:25.477085 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.473253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:49223 (local address 127.11.40.194:39583)
0929 00:55:25.473425 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.473430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.473671 (+ 241us) server_negotiation.cc:407] Connection header received
0929 00:55:25.473795 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.473798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.473856 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.473939 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:25.474267 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.474836 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476040 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.476435 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476536 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.476613 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.476858 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.476939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:55:25.477090 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.473419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:56321 (local address 127.11.40.195:33625)
0929 00:55:25.473547 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.473552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.473807 (+ 255us) server_negotiation.cc:407] Connection header received
0929 00:55:25.473945 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.473948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.473998 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.474080 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:25.474389 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.474871 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476029 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.476465 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476612 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.476664 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.476885 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.476955 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:55:25.477157 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.473217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:33625 (local address 127.11.40.193:56321)
0929 00:55:25.473682 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.473697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.473814 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.474101 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.474103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.474118 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:25.474247 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.474254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.475023 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.475026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.475901 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.475910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.476621 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.476721 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.476738 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.476866 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.476944 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":24,"thread_start_us":139,"threads_started":1}
I20260929 00:55:25.477116 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.473181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39583 (local address 127.11.40.193:49223)
0929 00:55:25.473573 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.473585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.473685 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.473967 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.473970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.473985 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:25.474138 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.474147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.474970 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.474976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.475901 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.475910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.476572 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.476576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.476685 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.476715 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:55:25.476779 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 00:55:25.476906 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":47,"thread_start_us":95,"threads_started":1}
I20260929 00:55:25.477931 29761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c0cb93710428aac10b07a2cb75573" candidate_uuid: "9f94b579975744b18492da0349b9949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54db8d94e0048d4b8106689f92651fe" is_pre_election: true
I20260929 00:55:25.477926 29893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c0cb93710428aac10b07a2cb75573" candidate_uuid: "9f94b579975744b18492da0349b9949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18da0337540943738f038e1bf4d24ac8" is_pre_election: true
I20260929 00:55:25.478252 29761 raft_consensus.cc:2454] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f94b579975744b18492da0349b9949f in term 0.
I20260929 00:55:25.478260 29893 raft_consensus.cc:2454] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f94b579975744b18492da0349b9949f in term 0.
I20260929 00:55:25.478796 29562 leader_election.cc:302] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18da0337540943738f038e1bf4d24ac8, 9f94b579975744b18492da0349b9949f, f54db8d94e0048d4b8106689f92651fe; no voters:
I20260929 00:55:25.479180 29963 raft_consensus.cc:2781] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:25.479290 29963 raft_consensus.cc:482] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:25.479337 29963 raft_consensus.cc:3037] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:25.480456 29963 raft_consensus.cc:504] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.480854 29963 leader_election.cc:288] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [CANDIDATE]: Term 1 election: Requested vote from peers f54db8d94e0048d4b8106689f92651fe (127.11.40.194:39583), 18da0337540943738f038e1bf4d24ac8 (127.11.40.195:33625)
I20260929 00:55:25.481350 29893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c0cb93710428aac10b07a2cb75573" candidate_uuid: "9f94b579975744b18492da0349b9949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18da0337540943738f038e1bf4d24ac8"
I20260929 00:55:25.481346 29761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c0cb93710428aac10b07a2cb75573" candidate_uuid: "9f94b579975744b18492da0349b9949f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54db8d94e0048d4b8106689f92651fe"
I20260929 00:55:25.481511 29893 raft_consensus.cc:3037] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:25.481483 29761 raft_consensus.cc:3037] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:25.482621 29893 raft_consensus.cc:2454] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f94b579975744b18492da0349b9949f in term 1.
I20260929 00:55:25.482630 29761 raft_consensus.cc:2454] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f94b579975744b18492da0349b9949f in term 1.
I20260929 00:55:25.483006 29565 leader_election.cc:302] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18da0337540943738f038e1bf4d24ac8, 9f94b579975744b18492da0349b9949f; no voters:
I20260929 00:55:25.483218 29963 raft_consensus.cc:2781] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:25.483623 29963 raft_consensus.cc:686] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 1 LEADER]: Becoming Leader. State: Replica: 9f94b579975744b18492da0349b9949f, State: Running, Role: LEADER
I20260929 00:55:25.484038 29963 consensus_queue.cc:237] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } }
I20260929 00:55:25.487718 29487 catalog_manager.cc:5495] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f94b579975744b18492da0349b9949f (127.11.40.193). New cstate: current_term: 1 leader_uuid: "9f94b579975744b18492da0349b9949f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f94b579975744b18492da0349b9949f" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 42669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:25.526134 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.522062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.11.40.193:42669)
0929 00:55:25.522235 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.522243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.522261 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:55:25.522384 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.522386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.522449 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.522583 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:25.522878 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.523442 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.524418 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.524662 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.524788 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.525162 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:25.525608 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:25.525786 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:25.525931 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 00:55:25.525972 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:55:25.532397 29893 raft_consensus.cc:1261] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 1 FOLLOWER]: Refusing update from remote peer 9f94b579975744b18492da0349b9949f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:25.532401 29761 raft_consensus.cc:1261] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 1 FOLLOWER]: Refusing update from remote peer 9f94b579975744b18492da0349b9949f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:25.533188 29963 consensus_queue.cc:1035] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54db8d94e0048d4b8106689f92651fe" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:25.533375 29968 consensus_queue.cc:1035] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18da0337540943738f038e1bf4d24ac8" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:25.553574 29486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:25.553782 29486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:25.556478 29486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3adc4171b274af3bdc5fc332588a2a7: Sending DeleteTablet for 3 replicas of tablet e02c0cb93710428aac10b07a2cb75573
I20260929 00:55:25.557310 29609 tablet_service.cc:1514] Processing DeleteTablet for tablet e02c0cb93710428aac10b07a2cb75573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:25 UTC) from {username='slave'} at 127.0.0.1:56752
I20260929 00:55:25.557349 29741 tablet_service.cc:1514] Processing DeleteTablet for tablet e02c0cb93710428aac10b07a2cb75573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:25 UTC) from {username='slave'} at 127.0.0.1:59676
I20260929 00:55:25.557854 29873 tablet_service.cc:1514] Processing DeleteTablet for tablet e02c0cb93710428aac10b07a2cb75573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:25 UTC) from {username='slave'} at 127.0.0.1:36746
I20260929 00:55:25.557955 29981 tablet_replica.cc:330] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f: stopping tablet replica
I20260929 00:55:25.558276 29981 raft_consensus.cc:2229] T e02c0cb93710428aac10b07a2cb75573 P 9f94b579975744b18492da0349b9949f [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:25.558472 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29547
I20260929 00:55:25.559341 29983 tablet_replica.cc:330] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: stopping tablet replica
I20260929 00:55:25.559882 29983 raft_consensus.cc:2229] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:25.560241 29983 raft_consensus.cc:2258] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:25.560555 29982 tablet_replica.cc:330] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: stopping tablet replica
I20260929 00:55:25.560894 29982 raft_consensus.cc:2229] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:25.561187 29982 raft_consensus.cc:2258] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:25.562495 29983 ts_tablet_manager.cc:1905] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:25.563464 29982 ts_tablet_manager.cc:1905] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:25.565528 29983 ts_tablet_manager.cc:1918] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:25.565662 29983 log.cc:1196] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/e02c0cb93710428aac10b07a2cb75573
I20260929 00:55:25.566053 29983 ts_tablet_manager.cc:1939] T e02c0cb93710428aac10b07a2cb75573 P 18da0337540943738f038e1bf4d24ac8: Deleting consensus metadata
I20260929 00:55:25.567183 29982 ts_tablet_manager.cc:1918] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:25.567287 29982 log.cc:1196] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/e02c0cb93710428aac10b07a2cb75573
I20260929 00:55:25.567449 29474 catalog_manager.cc:4841] TS 18da0337540943738f038e1bf4d24ac8 (127.11.40.195:33625): tablet e02c0cb93710428aac10b07a2cb75573 (table test-table [id=3fb94148564f49a8b7174e2cb79adf10]) successfully deleted
I20260929 00:55:25.567601 29982 ts_tablet_manager.cc:1939] T e02c0cb93710428aac10b07a2cb75573 P f54db8d94e0048d4b8106689f92651fe: Deleting consensus metadata
W20260929 00:55:25.567589 29473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:42669 (error 108)
I20260929 00:55:25.568506 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29678
I20260929 00:55:25.568711 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.568392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:42669 (local address 127.0.0.1:56772)
0929 00:55:25.568562 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.568629 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:42669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 00:55:25.569219 29473 catalog_manager.cc:4584] TS 9f94b579975744b18492da0349b9949f (127.11.40.193:42669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e02c0cb93710428aac10b07a2cb75573: Network error: Client connection negotiation failed: client connection to 127.11.40.193:42669: connect: Connection refused (error 111)
I20260929 00:55:25.574496 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29810
I20260929 00:55:25.574975 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.574613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39583 (local address 127.0.0.1:59680)
0929 00:55:25.574813 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.574872 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.194:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 00:55:25.580163 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 29456
2026-09-29T00:55:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1148 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:25 dist-test-slave-5309 krb5kdc[29987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:25 dist-test-slave-5309 krb5kdc[29987](info): set up 2 sockets
Sep 29 00:55:25 dist-test-slave-5309 krb5kdc[29987](info): commencing operation
krb5kdc: starting...
W20260929 00:55:26.650413 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:26 dist-test-slave-5309 krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:26.795697 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42981
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:26.913592 30000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:26.913911 30000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:26.913986 30000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:26.918397 30000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:26.918536 30000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:26.918581 30000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:26.918619 30000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:26.918654 30000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:26.924731 30000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:26.925983 30000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:26.927086 30000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:26.936283 30006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:26.936283 30005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:26.936563 30000 server_base.cc:1034] running on GCE node
W20260929 00:55:26.936582 30008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:26.937294 30000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:26.938570 30000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:26.939842 30000 hybrid_clock.cc:648] HybridClock initialized: now 1790643326939818 us; error 45 us; skew 500 ppm
Sep 29 00:55:26 dist-test-slave-5309 krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643326, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:26.943020 30000 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:26.944118 30000 webserver.cc:460] Webserver started at http://127.11.40.254:33545/ using document root <none> and password file <none>
I20260929 00:55:26.944749 30000 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:26.944890 30000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:26.945133 30000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:26.947273 30000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "128f16c4defa4c9db3802edfa3897853"
format_stamp: "Formatted at 2026-09-29 00:55:26 on dist-test-slave-5309"
server_key: "96ad8112cde575951d04cbca38d1801a"
server_key_iv: "5861edde384838ec2ae2a0645a322329"
server_key_version: "encryptionkey@0"
I20260929 00:55:26.947851 30000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "128f16c4defa4c9db3802edfa3897853"
format_stamp: "Formatted at 2026-09-29 00:55:26 on dist-test-slave-5309"
server_key: "96ad8112cde575951d04cbca38d1801a"
server_key_iv: "5861edde384838ec2ae2a0645a322329"
server_key_version: "encryptionkey@0"
I20260929 00:55:26.951769 30000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 00:55:26.954892 30015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:26.956390 30000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:26.956578 30000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "128f16c4defa4c9db3802edfa3897853"
format_stamp: "Formatted at 2026-09-29 00:55:26 on dist-test-slave-5309"
server_key: "96ad8112cde575951d04cbca38d1801a"
server_key_iv: "5861edde384838ec2ae2a0645a322329"
server_key_version: "encryptionkey@0"
I20260929 00:55:26.956720 30000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:26.978510 30000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:26.982120 30000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:26.982360 30000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:26.990172 30000 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:42981
I20260929 00:55:26.990229 30067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:42981 every 8 connection(s)
I20260929 00:55:26.991230 30000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:26.993505 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30000
I20260929 00:55:26.993626 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:26.993916 11427 external_mini_cluster.cc:1428] Setting key bc87ab38e7cf5fbf372ee1e012fbaa30
I20260929 00:55:26.994283 30068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:27.001255 30068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: Bootstrap starting.
Sep 29 00:55:26 dist-test-slave-5309 krb5kdc[29987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:27.004046 30068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:27.005066 30068 log.cc:824] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:27.007230 30068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: No bootstrap required, opened a new log
I20260929 00:55:27.008869 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:26.995468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.11.40.254:42981)
0929 00:55:26.995896 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:26.995911 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:26.995955 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:55:26.996611 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:26.996628 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:26.997028 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:26.997364 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:26.998169 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.999176 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.000059 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.000548 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.000719 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.003712 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.003717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.003731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.003761 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.005885 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.006374 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.006385 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.006390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.006506 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.006802 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.006806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.006808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.007225 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:27.007432 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.007731 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.008051 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260929 00:55:27.011483 30068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } }
I20260929 00:55:27.011775 30068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:27.011878 30068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128f16c4defa4c9db3802edfa3897853, State: Initialized, Role: FOLLOWER
I20260929 00:55:27.012422 30068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } }
I20260929 00:55:27.012640 30068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:27.012852 30068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:27.013238 30068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:27.014343 30068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } }
I20260929 00:55:27.014724 30068 leader_election.cc:302] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128f16c4defa4c9db3802edfa3897853; no voters:
I20260929 00:55:27.015086 30068 leader_election.cc:288] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:27.015232 30073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:27.015535 30073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [term 1 LEADER]: Becoming Leader. State: Replica: 128f16c4defa4c9db3802edfa3897853, State: Running, Role: LEADER
I20260929 00:55:27.015902 30073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } }
I20260929 00:55:27.016693 30068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:27.017515 30075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128f16c4defa4c9db3802edfa3897853. Latest consensus state: current_term: 1 leader_uuid: "128f16c4defa4c9db3802edfa3897853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } } }
I20260929 00:55:27.017653 30075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:27.017915 30074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128f16c4defa4c9db3802edfa3897853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128f16c4defa4c9db3802edfa3897853" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 42981 } } }
I20260929 00:55:27.018020 30074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853 [sys.catalog]: This master's current role is: LEADER
W20260929 00:55:27.022747 30089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:55:27.022873 30089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:55:27.023051 30082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:27.023869 30082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:27.030867 30082 catalog_manager.cc:1347] Generated new cluster ID: ed9c5d9f7d324c8cb0911ada99a421f8
I20260929 00:55:27.030967 30082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:27.051218 30082 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:27.052201 30082 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:27.060410 30082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: Generated new TSK 0
I20260929 00:55:27.061352 30082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
Entry for principal kudu/127.11.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab.
I20260929 00:55:27.105720 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:27.238302 30095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:27.238615 30095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:27.238664 30095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:27.243155 30095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:27.243233 30095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:27.243299 30095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:27.249136 30095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.193.keytab
--principal=kudu/127.11.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:27.250229 30095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:27.251284 30095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:27.261574 30100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:27.261588 30101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:27.261603 30103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:27.262189 30095 server_base.cc:1034] running on GCE node
I20260929 00:55:27.262706 30095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:27.263527 30095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:27.264756 30095 hybrid_clock.cc:648] HybridClock initialized: now 1790643327264712 us; error 51 us; skew 500 ppm
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:27.267884 30095 init.cc:377] Logged in from keytab as kudu/127.11.40.193@KRBTEST.COM (short username kudu)
I20260929 00:55:27.268936 30095 webserver.cc:460] Webserver started at http://127.11.40.193:41471/ using document root <none> and password file <none>
I20260929 00:55:27.269625 30095 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:27.269737 30095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:27.269932 30095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:27.272089 30095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "6155befa65b74b51b0549c2d3eec24bf"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "1546c0647e5e2657190b57a458cf37b9"
server_key_iv: "38470aca6543c0d7eefa8fd8ecba03df"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.272644 30095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "6155befa65b74b51b0549c2d3eec24bf"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "1546c0647e5e2657190b57a458cf37b9"
server_key_iv: "38470aca6543c0d7eefa8fd8ecba03df"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.276639 30095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260929 00:55:27.279471 30110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.281277 30095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:27.281450 30095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "6155befa65b74b51b0549c2d3eec24bf"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "1546c0647e5e2657190b57a458cf37b9"
server_key_iv: "38470aca6543c0d7eefa8fd8ecba03df"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.281594 30095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:27.309598 30095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:27.313798 30095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:27.314040 30095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:27.314733 30095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:27.315681 30095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:27.315761 30095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.315845 30095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:27.315903 30095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.326493 30095 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:33909
I20260929 00:55:27.326508 30223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:33909 every 8 connection(s)
I20260929 00:55:27.327533 30095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:27.334173 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30095
I20260929 00:55:27.334324 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:27.334628 11427 external_mini_cluster.cc:1428] Setting key 3f6cea4e54740c7d33217d8e72e51d93
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.193@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:27.344497 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.329543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:36329 (local address 127.11.40.254:42981)
0929 00:55:27.329737 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.329742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.330739 (+ 997us) server_negotiation.cc:407] Connection header received
0929 00:55:27.332134 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.332138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.332213 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.332298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.333714 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.334528 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.335909 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.336368 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.336485 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.339979 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.339982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.339984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.340011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.341984 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.342577 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.342584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.342588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.342696 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.343085 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.343090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.343091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.343359 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:27.343494 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.344122 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.344281 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.11.40.194@KRBTEST.COM; defaulting to no policy
I20260929 00:55:27.345736 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.329988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42981 (local address 127.11.40.193:36329)
0929 00:55:27.330554 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.330603 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.331871 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.332538 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.332546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.333033 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:27.333548 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.333563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.334662 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.334665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.335769 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.335779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.336561 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.336567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.336785 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.337512 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:27.337538 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:27.339770 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:27.342139 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.342145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.342159 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.342458 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.342810 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.342828 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.342829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.342978 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.343526 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:27.343533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:27.343837 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.344029 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.344393 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":127,"threads_started":1}
I20260929 00:55:27.347471 30224 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42981
I20260929 00:55:27.347728 30224 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:27.348374 30224 heartbeater.cc:511] Master 127.11.40.254:42981 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.194@KRBTEST.COM" created.
I20260929 00:55:27.350322 30032 ts_manager.cc:194] Registered new tserver with Master: 6155befa65b74b51b0549c2d3eec24bf (127.11.40.193:33909)
I20260929 00:55:27.351653 30032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.193@KRBTEST.COM'} at 127.11.40.193:36329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
Entry for principal kudu/127.11.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab.
I20260929 00:55:27.368731 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:27.484133 30229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:27.484476 30229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:27.484558 30229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:27.488978 30229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:27.489085 30229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:27.489187 30229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:27.495108 30229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.194.keytab
--principal=kudu/127.11.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:27.496656 30229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:27.497956 30229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:27.508745 30235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:27.508762 30234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:27.509087 30229 server_base.cc:1034] running on GCE node
W20260929 00:55:27.509151 30237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:27.509657 30229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:27.510356 30229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:27.511549 30229 hybrid_clock.cc:648] HybridClock initialized: now 1790643327511532 us; error 30 us; skew 500 ppm
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:27.514357 30229 init.cc:377] Logged in from keytab as kudu/127.11.40.194@KRBTEST.COM (short username kudu)
I20260929 00:55:27.515285 30229 webserver.cc:460] Webserver started at http://127.11.40.194:38811/ using document root <none> and password file <none>
I20260929 00:55:27.515874 30229 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:27.515969 30229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:27.516201 30229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:27.518519 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "9776289aa5d54b4ca28f781845d7504c"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "b4c6f8b28be490cbf24fd7b7b1a45f0f"
server_key_iv: "f913c4fe5bb8454a38b7c4e8bf0b4d36"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.519135 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "9776289aa5d54b4ca28f781845d7504c"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "b4c6f8b28be490cbf24fd7b7b1a45f0f"
server_key_iv: "f913c4fe5bb8454a38b7c4e8bf0b4d36"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.523147 30229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:27.525705 30245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.527021 30229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:27.527168 30229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "9776289aa5d54b4ca28f781845d7504c"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "b4c6f8b28be490cbf24fd7b7b1a45f0f"
server_key_iv: "f913c4fe5bb8454a38b7c4e8bf0b4d36"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.527294 30229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:27.545948 30229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:27.549360 30229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:27.549629 30229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:27.550452 30229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:27.551415 30229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:27.551494 30229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.551575 30229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:27.551621 30229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.562477 30229 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:42759
I20260929 00:55:27.562530 30359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:42759 every 8 connection(s)
I20260929 00:55:27.563455 30229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:27.566280 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30229
I20260929 00:55:27.566437 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:27.566751 11427 external_mini_cluster.cc:1428] Setting key 9eecd298a1cebae1d865fd9d9b8e7525
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.194@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.195@KRBTEST.COM; defaulting to no policy
I20260929 00:55:27.581246 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.565462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:56577 (local address 127.11.40.254:42981)
0929 00:55:27.565624 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.565629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.566546 (+ 917us) server_negotiation.cc:407] Connection header received
0929 00:55:27.568227 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.568230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.568278 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.568354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.570148 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.571150 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.572993 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.573310 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.573413 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.576911 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.576915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.576918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.576959 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.578867 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.579544 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.579547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.579549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.579637 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.579976 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.579978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.579979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.580180 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:27.580260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.580960 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.581085 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.195@KRBTEST.COM" created.
I20260929 00:55:27.582290 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.565892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42981 (local address 127.11.40.194:56577)
0929 00:55:27.566385 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.566440 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.568010 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.568546 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.568553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.569022 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:27.569818 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.569843 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.571321 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.571324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.572793 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.572852 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.573509 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.573516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.573785 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.574577 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:27.574602 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:27.576632 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:27.579024 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.579031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.579046 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.579432 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.579727 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.579729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.579730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.579887 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.580304 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:27.580310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:27.580641 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.580890 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.581306 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260929 00:55:27.583606 30360 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42981
I20260929 00:55:27.583887 30360 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:27.584419 30360 heartbeater.cc:511] Master 127.11.40.254:42981 requested a full tablet report, sending...
I20260929 00:55:27.585634 30032 ts_manager.cc:194] Registered new tserver with Master: 9776289aa5d54b4ca28f781845d7504c (127.11.40.194:42759)
I20260929 00:55:27.586253 30032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.194@KRBTEST.COM'} at 127.11.40.194:56577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
Entry for principal kudu/127.11.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab.
I20260929 00:55:27.604617 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:27.725200 30365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:27.725555 30365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:27.725644 30365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:27.729892 30365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:27.730006 30365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:27.730100 30365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:27.735927 30365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.195.keytab
--principal=kudu/127.11.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:27.737151 30365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:27.738214 30365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:27.748350 30370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:27.748375 30373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:27.748368 30371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:27.748554 30365 server_base.cc:1034] running on GCE node
I20260929 00:55:27.749186 30365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:27.749886 30365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:27.751055 30365 hybrid_clock.cc:648] HybridClock initialized: now 1790643327751037 us; error 28 us; skew 500 ppm
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:27.754199 30365 init.cc:377] Logged in from keytab as kudu/127.11.40.195@KRBTEST.COM (short username kudu)
I20260929 00:55:27.755200 30365 webserver.cc:460] Webserver started at http://127.11.40.195:34897/ using document root <none> and password file <none>
I20260929 00:55:27.755836 30365 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:27.755937 30365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:27.756189 30365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:27.758440 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "9aa8ebf017694ac7bfba8ac4017e743f"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "8a3bade61208fb0957728619c0740f90"
server_key_iv: "232e655581db4a42a07496c7e0c5f125"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.759069 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "9aa8ebf017694ac7bfba8ac4017e743f"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "8a3bade61208fb0957728619c0740f90"
server_key_iv: "232e655581db4a42a07496c7e0c5f125"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.763404 30365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:27.766273 30381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.767608 30365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:27.767757 30365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "9aa8ebf017694ac7bfba8ac4017e743f"
format_stamp: "Formatted at 2026-09-29 00:55:27 on dist-test-slave-5309"
server_key: "8a3bade61208fb0957728619c0740f90"
server_key_iv: "232e655581db4a42a07496c7e0c5f125"
server_key_version: "encryptionkey@0"
I20260929 00:55:27.767911 30365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:27.778489 30365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:27.782090 30365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:27.782322 30365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:27.782954 30365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:27.783829 30365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:27.783898 30365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.783970 30365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:27.784011 30365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:27.795480 30365 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:35293
I20260929 00:55:27.795555 30494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:35293 every 8 connection(s)
I20260929 00:55:27.796583 30365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:27.803768 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30365
I20260929 00:55:27.803941 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:27.804250 11427 external_mini_cluster.cc:1428] Setting key a01187cc3822d1237d58ac33ea5e25ba
Sep 29 00:55:27 dist-test-slave-5309 krb5kdc[29987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643327, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.195@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:27.811347 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.798638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:52997 (local address 127.11.40.254:42981)
0929 00:55:27.798798 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.798803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.799629 (+ 826us) server_negotiation.cc:407] Connection header received
0929 00:55:27.801124 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.801127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.801195 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.801288 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.802609 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.803216 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.804708 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.804991 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.805115 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.807884 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.807892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.807894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.807919 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.809307 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.809815 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.809827 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.809833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.809908 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.810251 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.810256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.810258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.810446 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:27.810553 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.811056 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.811183 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:55:27.812223 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.799009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:42981 (local address 127.11.40.195:52997)
0929 00:55:27.799468 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.799514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.800885 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.801498 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.801506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.801912 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:27.802441 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.802455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.803374 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.803385 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.804534 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.804545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.805156 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.805160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.805374 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.806137 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:27.806154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:27.807707 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:27.809432 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.809438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.809453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.809708 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.810008 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:27.810011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:27.810013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:27.810154 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:27.810551 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:27.810558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:27.810808 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.811016 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.811364 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260929 00:55:27.813539 30495 heartbeater.cc:347] Connected to a master server at 127.11.40.254:42981
I20260929 00:55:27.813827 30495 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:27.814378 30495 heartbeater.cc:511] Master 127.11.40.254:42981 requested a full tablet report, sending...
I20260929 00:55:27.815604 30032 ts_manager.cc:194] Registered new tserver with Master: 9aa8ebf017694ac7bfba8ac4017e743f (127.11.40.195:35293)
I20260929 00:55:27.816455 30032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.40.195@KRBTEST.COM'} at 127.11.40.195:52997
I20260929 00:55:27.818607 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:27.828912 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.821030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.11.40.254:42981)
0929 00:55:27.821367 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.821373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.821492 (+ 119us) server_negotiation.cc:407] Connection header received
0929 00:55:27.821663 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.821673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.821722 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.821824 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.822578 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.823104 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.824035 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.824263 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.824380 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.825912 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.825916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.825919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.825953 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.827145 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.827583 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.827587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.827588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.827675 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:27.827973 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:27.827978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:27.827980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:27.828211 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:27.828365 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.828601 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.828722 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260929 00:55:27.832458 30032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:27.835624 30032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:27.850863 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.844617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:35293 (local address 127.0.0.1:42028)
0929 00:55:27.845382 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.845396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.845485 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.846461 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.846464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.846484 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.846685 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.846693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.847954 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.847959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.849280 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.849291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850290 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.850397 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.850434 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.850581 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.850708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":51,"thread_start_us":176,"threads_started":1}
I20260929 00:55:27.850871 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.845444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.11.40.195:35293)
0929 00:55:27.845957 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.845973 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.845994 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:55:27.846067 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.846072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.846303 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.846653 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.846779 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.847810 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.849422 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850147 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850283 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.850436 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.850615 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.850711 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":172,"threads_started":1}
I20260929 00:55:27.851334 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.844610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:42759 (local address 127.0.0.1:35132)
0929 00:55:27.845021 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.845071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.845202 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.846429 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.846436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.846463 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.846832 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.846839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.848489 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.848492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.849646 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.849656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850879 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.851012 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.851045 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.851118 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.851186 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260929 00:55:27.851722 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.845912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.11.40.193:33909)
0929 00:55:27.846433 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.846448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.846469 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:55:27.846688 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.846693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.846874 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.847014 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.847275 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.848366 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.849720 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850571 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.850809 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.851489 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.851601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":64,"threads_started":1}
I20260929 00:55:27.852072 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.845448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33909 (local address 127.0.0.1:57948)
0929 00:55:27.845679 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.845693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.846538 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.847033 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.847036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.847055 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.847152 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.847158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.848512 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.848515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.849586 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.849595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850879 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.851150 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.851168 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.851228 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.851282 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":22,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
I20260929 00:55:27.852099 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.844960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.11.40.194:42759)
0929 00:55:27.845882 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.845896 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.845921 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:55:27.846015 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.846020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.846240 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.846976 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.847138 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.848332 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.849761 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.850571 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.850680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.850809 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.851866 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.851967 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":113,"threads_started":1}
I20260929 00:55:27.854480 30294 tablet_service.cc:1467] Processing CreateTablet for tablet 7e22d471c99141bc885394f169324381 (DEFAULT_TABLE table=test-table [id=9beaa91991a9419e897d579f14f4121e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:27.854385 30429 tablet_service.cc:1467] Processing CreateTablet for tablet 7e22d471c99141bc885394f169324381 (DEFAULT_TABLE table=test-table [id=9beaa91991a9419e897d579f14f4121e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:27.855468 30294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e22d471c99141bc885394f169324381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:27.855508 30429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e22d471c99141bc885394f169324381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:27.855165 30158 tablet_service.cc:1467] Processing CreateTablet for tablet 7e22d471c99141bc885394f169324381 (DEFAULT_TABLE table=test-table [id=9beaa91991a9419e897d579f14f4121e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:27.856092 30158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e22d471c99141bc885394f169324381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:27.862725 30512 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Bootstrap starting.
I20260929 00:55:27.863972 30515 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Bootstrap starting.
I20260929 00:55:27.865167 30512 tablet_bootstrap.cc:654] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:27.865159 30514 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: Bootstrap starting.
I20260929 00:55:27.866074 30515 tablet_bootstrap.cc:654] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:27.866335 30512 log.cc:824] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:27.867271 30515 log.cc:824] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:27.867364 30514 tablet_bootstrap.cc:654] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:27.868134 30514 log.cc:824] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:27.869067 30512 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: No bootstrap required, opened a new log
I20260929 00:55:27.869284 30512 ts_tablet_manager.cc:1397] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260929 00:55:27.869372 30515 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: No bootstrap required, opened a new log
I20260929 00:55:27.869561 30515 ts_tablet_manager.cc:1397] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 00:55:27.870100 30514 tablet_bootstrap.cc:492] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: No bootstrap required, opened a new log
I20260929 00:55:27.870285 30514 ts_tablet_manager.cc:1397] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:55:27.873333 30512 raft_consensus.cc:348] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.873669 30512 raft_consensus.cc:374] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:27.873728 30512 raft_consensus.cc:729] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9776289aa5d54b4ca28f781845d7504c, State: Initialized, Role: FOLLOWER
I20260929 00:55:27.873786 30515 raft_consensus.cc:348] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.874061 30515 raft_consensus.cc:374] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:27.874110 30515 raft_consensus.cc:729] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa8ebf017694ac7bfba8ac4017e743f, State: Initialized, Role: FOLLOWER
I20260929 00:55:27.874593 30515 consensus_queue.cc:260] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.874840 30512 consensus_queue.cc:260] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.874962 30514 raft_consensus.cc:348] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.875226 30514 raft_consensus.cc:374] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:27.875305 30514 raft_consensus.cc:729] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6155befa65b74b51b0549c2d3eec24bf, State: Initialized, Role: FOLLOWER
I20260929 00:55:27.875762 30515 ts_tablet_manager.cc:1428] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:27.875727 30514 consensus_queue.cc:260] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.875818 30360 heartbeater.cc:503] Master 127.11.40.254:42981 was elected leader, sending a full tablet report...
I20260929 00:55:27.875900 30512 ts_tablet_manager.cc:1428] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260929 00:55:27.876124 30495 heartbeater.cc:503] Master 127.11.40.254:42981 was elected leader, sending a full tablet report...
I20260929 00:55:27.876652 30514 ts_tablet_manager.cc:1428] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:27.876830 30224 heartbeater.cc:503] Master 127.11.40.254:42981 was elected leader, sending a full tablet report...
I20260929 00:55:27.953430 30520 raft_consensus.cc:482] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:27.953699 30520 raft_consensus.cc:504] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.954988 30520 leader_election.cc:288] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aa8ebf017694ac7bfba8ac4017e743f (127.11.40.195:35293), 6155befa65b74b51b0549c2d3eec24bf (127.11.40.193:33909)
I20260929 00:55:27.957888 30519 raft_consensus.cc:482] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:27.958110 30519 raft_consensus.cc:504] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.959081 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.955335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:51549 (local address 127.11.40.195:35293)
0929 00:55:27.955491 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.955498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.955520 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:55:27.955706 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.955709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.955753 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.955850 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.956134 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.956732 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.957886 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.958495 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.958629 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.958685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.958885 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.958959 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:55:27.959081 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.955193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:35293 (local address 127.11.40.194:51549)
0929 00:55:27.955418 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.955436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.955539 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.955865 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.955868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.955888 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.956015 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.956021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.956880 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.956885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.957725 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.957734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.958629 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.958632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.958741 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.958778 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.958868 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.958948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 00:55:27.959528 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.955470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:47743 (local address 127.11.40.193:33909)
0929 00:55:27.955605 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.955610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.955831 (+ 221us) server_negotiation.cc:407] Connection header received
0929 00:55:27.955990 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.955993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.956050 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.956138 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.956416 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.956982 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.958214 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.958828 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.958947 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.958995 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.959306 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.959394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:55:27.959898 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.955372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33909 (local address 127.11.40.194:47743)
0929 00:55:27.955716 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.955727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.955871 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.956148 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.956151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.956166 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.956298 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.956304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.957088 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.957094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.958076 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.958086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.959058 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.959062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.959157 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.959179 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.959667 (+ 488us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.959746 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260929 00:55:27.959877 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9776289aa5d54b4ca28f781845d7504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" is_pre_election: true
I20260929 00:55:27.960153 30449 raft_consensus.cc:2454] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9776289aa5d54b4ca28f781845d7504c in term 0.
I20260929 00:55:27.960678 30249 leader_election.cc:302] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9776289aa5d54b4ca28f781845d7504c, 9aa8ebf017694ac7bfba8ac4017e743f; no voters:
I20260929 00:55:27.960924 30178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9776289aa5d54b4ca28f781845d7504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6155befa65b74b51b0549c2d3eec24bf" is_pre_election: true
I20260929 00:55:27.961019 30520 raft_consensus.cc:2781] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:27.961098 30520 raft_consensus.cc:482] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:27.961143 30520 raft_consensus.cc:3037] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:27.961231 30178 raft_consensus.cc:2454] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9776289aa5d54b4ca28f781845d7504c in term 0.
I20260929 00:55:27.959906 30519 leader_election.cc:288] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9776289aa5d54b4ca28f781845d7504c (127.11.40.194:42759), 6155befa65b74b51b0549c2d3eec24bf (127.11.40.193:33909)
I20260929 00:55:27.962536 30520 raft_consensus.cc:504] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.962908 30520 leader_election.cc:288] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [CANDIDATE]: Term 1 election: Requested vote from peers 9aa8ebf017694ac7bfba8ac4017e743f (127.11.40.195:35293), 6155befa65b74b51b0549c2d3eec24bf (127.11.40.193:33909)
I20260929 00:55:27.963243 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9776289aa5d54b4ca28f781845d7504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa8ebf017694ac7bfba8ac4017e743f"
I20260929 00:55:27.963344 30178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9776289aa5d54b4ca28f781845d7504c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6155befa65b74b51b0549c2d3eec24bf"
I20260929 00:55:27.963380 30449 raft_consensus.cc:3037] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:27.963474 30178 raft_consensus.cc:3037] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:27.964556 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.959627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:42759 (local address 127.11.40.195:45583)
0929 00:55:27.960009 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.960023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.960158 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.960865 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.960868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.960883 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.960988 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.960993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.961896 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.961899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.963123 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.963134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.964099 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.964103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.964215 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.964240 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.964322 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.964402 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":5}
I20260929 00:55:27.964720 30449 raft_consensus.cc:2454] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9776289aa5d54b4ca28f781845d7504c in term 1.
I20260929 00:55:27.964861 30178 raft_consensus.cc:2454] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9776289aa5d54b4ca28f781845d7504c in term 1.
I20260929 00:55:27.965044 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.959704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:45583 (local address 127.11.40.194:42759)
0929 00:55:27.959801 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.959805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.960425 (+ 620us) server_negotiation.cc:407] Connection header received
0929 00:55:27.960472 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.960475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.960532 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.960598 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.961088 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.961797 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.963254 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.963877 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.963989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.964037 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.964854 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.964935 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:55:27.965088 30249 leader_election.cc:302] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9776289aa5d54b4ca28f781845d7504c, 9aa8ebf017694ac7bfba8ac4017e743f; no voters:
I20260929 00:55:27.965279 30520 raft_consensus.cc:2781] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:27.965548 30314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9776289aa5d54b4ca28f781845d7504c" is_pre_election: true
I20260929 00:55:27.965730 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.960129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33909 (local address 127.11.40.195:41979)
0929 00:55:27.960490 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.960504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.960598 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.961486 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.961489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.961504 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:27.961609 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.961615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.962518 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.962520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.963949 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.963959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.965295 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.965298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.965405 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.965427 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:55:27.965508 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:55:27.965587 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":47,"thread_start_us":166,"threads_started":1}
I20260929 00:55:27.965776 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.960367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:41979 (local address 127.11.40.193:33909)
0929 00:55:27.961168 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.961172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.961183 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:55:27.961219 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.961222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.961261 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.961358 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:27.961832 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.962403 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.964091 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.964909 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.964994 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.965046 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:27.965555 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 00:55:27.965643 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722}
I20260929 00:55:27.965602 30520 raft_consensus.cc:686] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 1 LEADER]: Becoming Leader. State: Replica: 9776289aa5d54b4ca28f781845d7504c, State: Running, Role: LEADER
I20260929 00:55:27.966382 30178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e22d471c99141bc885394f169324381" candidate_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6155befa65b74b51b0549c2d3eec24bf" is_pre_election: true
I20260929 00:55:27.966276 30520 consensus_queue.cc:237] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } }
I20260929 00:55:27.966562 30178 raft_consensus.cc:2379] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9aa8ebf017694ac7bfba8ac4017e743f in current term 1: Already voted for candidate 9776289aa5d54b4ca28f781845d7504c in this term.
I20260929 00:55:27.969611 30382 leader_election.cc:302] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9aa8ebf017694ac7bfba8ac4017e743f; no voters: 6155befa65b74b51b0549c2d3eec24bf, 9776289aa5d54b4ca28f781845d7504c
I20260929 00:55:27.969941 30519 raft_consensus.cc:2726] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 00:55:27.970052 30032 catalog_manager.cc:5495] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9776289aa5d54b4ca28f781845d7504c (127.11.40.194). New cstate: current_term: 1 leader_uuid: "9776289aa5d54b4ca28f781845d7504c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9776289aa5d54b4ca28f781845d7504c" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 42759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:28.007907 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.003474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.11.40.194:42759)
0929 00:55:28.003643 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.003648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.003666 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:55:28.003765 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.003768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.003840 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.003963 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:28.004229 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.004934 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.005967 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.006200 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.006306 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.006772 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:28.007343 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:28.007580 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.007621 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.007685 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:55:28.015026 30178 raft_consensus.cc:1261] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 1 FOLLOWER]: Refusing update from remote peer 9776289aa5d54b4ca28f781845d7504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:28.015092 30449 raft_consensus.cc:1261] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 1 FOLLOWER]: Refusing update from remote peer 9776289aa5d54b4ca28f781845d7504c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:28.015993 30520 consensus_queue.cc:1035] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa8ebf017694ac7bfba8ac4017e743f" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:28.016229 30524 consensus_queue.cc:1035] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6155befa65b74b51b0549c2d3eec24bf" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:28.036212 30032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:28.036441 30032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:28.039362 30032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 128f16c4defa4c9db3802edfa3897853: Sending DeleteTablet for 3 replicas of tablet 7e22d471c99141bc885394f169324381
I20260929 00:55:28.040310 30158 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e22d471c99141bc885394f169324381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:28 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:57948
I20260929 00:55:28.040287 30429 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e22d471c99141bc885394f169324381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:28 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:42028
I20260929 00:55:28.040922 30537 tablet_replica.cc:330] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf: stopping tablet replica
I20260929 00:55:28.041083 30294 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e22d471c99141bc885394f169324381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:28 UTC) from {username='kudu', principal='kudu/127.11.40.254@KRBTEST.COM'} at 127.0.0.1:35132
I20260929 00:55:28.041162 30538 tablet_replica.cc:330] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: stopping tablet replica
I20260929 00:55:28.041447 30538 raft_consensus.cc:2229] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:28.041448 30537 raft_consensus.cc:2229] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:28.041792 30538 raft_consensus.cc:2258] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:28.041923 30537 raft_consensus.cc:2258] T 7e22d471c99141bc885394f169324381 P 6155befa65b74b51b0549c2d3eec24bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:28.042137 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30095
I20260929 00:55:28.043767 30539 tablet_replica.cc:330] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: stopping tablet replica
I20260929 00:55:28.044080 30539 raft_consensus.cc:2229] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:28.044492 30539 raft_consensus.cc:2258] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:28.046097 30539 ts_tablet_manager.cc:1905] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:28.048834 30538 ts_tablet_manager.cc:1905] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:28.049027 30539 ts_tablet_manager.cc:1918] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:28.049135 30539 log.cc:1196] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/7e22d471c99141bc885394f169324381
I20260929 00:55:28.049455 30539 ts_tablet_manager.cc:1939] T 7e22d471c99141bc885394f169324381 P 9776289aa5d54b4ca28f781845d7504c: Deleting consensus metadata
I20260929 00:55:28.051451 30016 catalog_manager.cc:4841] TS 9776289aa5d54b4ca28f781845d7504c (127.11.40.194:42759): tablet 7e22d471c99141bc885394f169324381 (table test-table [id=9beaa91991a9419e897d579f14f4121e]) successfully deleted
W20260929 00:55:28.051772 30016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:33909 (error 108)
I20260929 00:55:28.051810 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30229
I20260929 00:55:28.053180 30538 ts_tablet_manager.cc:1918] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:28.053375 30538 log.cc:1196] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/7e22d471c99141bc885394f169324381
I20260929 00:55:28.053772 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.053494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:33909 (local address 127.0.0.1:57954)
0929 00:55:28.053606 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.053680 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 00:55:28.053982 30016 catalog_manager.cc:4584] TS 6155befa65b74b51b0549c2d3eec24bf (127.11.40.193:33909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e22d471c99141bc885394f169324381: Network error: Client connection negotiation failed: client connection to 127.11.40.193:33909: connect: Connection refused (error 111)
I20260929 00:55:28.054134 30538 ts_tablet_manager.cc:1939] T 7e22d471c99141bc885394f169324381 P 9aa8ebf017694ac7bfba8ac4017e743f: Deleting consensus metadata
I20260929 00:55:28.055540 30019 catalog_manager.cc:4841] TS 9aa8ebf017694ac7bfba8ac4017e743f (127.11.40.195:35293): tablet 7e22d471c99141bc885394f169324381 (table test-table [id=9beaa91991a9419e897d579f14f4121e]) successfully deleted
I20260929 00:55:28.058801 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30365
I20260929 00:55:28.065347 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30000
2026-09-29T00:55:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2486 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T00:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:28Z Disabled control of system clock
I20260929 00:55:28.097419 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39651
--webserver_interface=127.11.40.254
--webserver_port=0
--builtin_ntp_servers=127.11.40.212:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:28.220384 30544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.220672 30544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.220708 30544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.225236 30544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:28.225329 30544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.225346 30544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.225366 30544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:28.225382 30544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:28.230489 30544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:39651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:28.231622 30544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.233021 30544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:28.241513 30550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.241652 30549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.241541 30552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.241770 30544 server_base.cc:1034] running on GCE node
I20260929 00:55:28.242422 30544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:28.243610 30544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:28.244854 30544 hybrid_clock.cc:648] HybridClock initialized: now 1790643328244798 us; error 39 us; skew 500 ppm
I20260929 00:55:28.246852 30544 webserver.cc:460] Webserver started at http://127.11.40.254:38007/ using document root <none> and password file <none>
I20260929 00:55:28.247468 30544 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:28.247567 30544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:28.247818 30544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:28.250061 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "5140b52880a54412ab08d3798d63316a"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4fab8bf6c4793504ebabbb655902b201"
server_key_iv: "e5d0840e390b719a5590c1605c07e618"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.250651 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "5140b52880a54412ab08d3798d63316a"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4fab8bf6c4793504ebabbb655902b201"
server_key_iv: "e5d0840e390b719a5590c1605c07e618"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.254489 30544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:55:28.256994 30558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.258416 30544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:28.258576 30544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "5140b52880a54412ab08d3798d63316a"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4fab8bf6c4793504ebabbb655902b201"
server_key_iv: "e5d0840e390b719a5590c1605c07e618"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.258698 30544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:28.278513 30544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:28.279258 30544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:28.279469 30544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:28.287575 30544 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:39651
I20260929 00:55:28.287604 30610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:39651 every 8 connection(s)
I20260929 00:55:28.288968 30544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:28.293110 30611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:28.294857 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30544
I20260929 00:55:28.294988 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:28.295289 11427 external_mini_cluster.cc:1428] Setting key 6581a1dcee531f2ec181914f7328982b
I20260929 00:55:28.299731 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: Bootstrap starting.
I20260929 00:55:28.302081 30611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:28.302837 30611 log.cc:824] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:28.304788 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.296840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.11.40.254:39651)
0929 00:55:28.297472 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.297488 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.297524 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:55:28.298172 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.298189 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.298591 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.298932 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.299361 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.300381 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.301327 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.301804 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.301979 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.302860 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.302866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.302878 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.303374 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.303445 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.303472 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.303577 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.303778 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.304062 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260929 00:55:28.305037 30611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: No bootstrap required, opened a new log
I20260929 00:55:28.308683 30611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } }
I20260929 00:55:28.308997 30611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:28.309082 30611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5140b52880a54412ab08d3798d63316a, State: Initialized, Role: FOLLOWER
I20260929 00:55:28.309540 30611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } }
I20260929 00:55:28.309693 30611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:28.309762 30611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:28.309862 30611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:28.310850 30611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } }
I20260929 00:55:28.311190 30611 leader_election.cc:302] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5140b52880a54412ab08d3798d63316a; no voters:
I20260929 00:55:28.311523 30611 leader_election.cc:288] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:28.311769 30616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:28.312031 30616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [term 1 LEADER]: Becoming Leader. State: Replica: 5140b52880a54412ab08d3798d63316a, State: Running, Role: LEADER
I20260929 00:55:28.312731 30616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } }
I20260929 00:55:28.312850 30611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:28.314574 30618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5140b52880a54412ab08d3798d63316a. Latest consensus state: current_term: 1 leader_uuid: "5140b52880a54412ab08d3798d63316a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } } }
I20260929 00:55:28.314697 30618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:28.314962 30617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5140b52880a54412ab08d3798d63316a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5140b52880a54412ab08d3798d63316a" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39651 } } }
I20260929 00:55:28.315845 30617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:28.315253 30630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:28.318519 30630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:28.324604 30630 catalog_manager.cc:1347] Generated new cluster ID: c5ff59a254c440b98c27ec65dcbfdf06
I20260929 00:55:28.324699 30630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:28.330067 30630 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:28.331110 30630 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:28.335667 30630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: Generated new TSK 0
I20260929 00:55:28.336354 30630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:28.343621 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.193:0
--local_ip_for_outbound_sockets=127.11.40.193
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--builtin_ntp_servers=127.11.40.212:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:55:28.458434 30635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.458696 30635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.458736 30635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.463300 30635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.463369 30635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.463462 30635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.193
I20260929 00:55:28.468554 30635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.40.193
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.193
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:28.469800 30635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.471275 30635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:28.481587 30640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.481710 30635 server_base.cc:1034] running on GCE node
W20260929 00:55:28.481609 30643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.481546 30641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.482285 30635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:28.483021 30635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:28.484280 30635 hybrid_clock.cc:648] HybridClock initialized: now 1790643328484242 us; error 40 us; skew 500 ppm
I20260929 00:55:28.486454 30635 webserver.cc:460] Webserver started at http://127.11.40.193:42079/ using document root <none> and password file <none>
I20260929 00:55:28.487112 30635 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:28.487212 30635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:28.487484 30635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:28.489818 30635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data/instance:
uuid: "2ac6556b87ce40989e90e573dc7acc60"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "de1494c3d1430c99b8c34b399041e446"
server_key_iv: "67a6f8a5135669895ba64111c15e44b7"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.490406 30635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance:
uuid: "2ac6556b87ce40989e90e573dc7acc60"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "de1494c3d1430c99b8c34b399041e446"
server_key_iv: "67a6f8a5135669895ba64111c15e44b7"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.494338 30635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:28.497251 30650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.498678 30635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:28.498852 30635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal
uuid: "2ac6556b87ce40989e90e573dc7acc60"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "de1494c3d1430c99b8c34b399041e446"
server_key_iv: "67a6f8a5135669895ba64111c15e44b7"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.498991 30635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:28.512436 30635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:28.513392 30635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:28.513603 30635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:28.514495 30635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:28.515707 30635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:28.515792 30635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.515867 30635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:28.515902 30635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.525854 30635 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.193:45337
I20260929 00:55:28.525875 30763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.193:45337 every 8 connection(s)
I20260929 00:55:28.526902 30635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:28.530633 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30635
I20260929 00:55:28.530787 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:28.531112 11427 external_mini_cluster.cc:1428] Setting key f43ebee9fb6926b392e96113ba6bce6c
I20260929 00:55:28.533425 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.194:0
--local_ip_for_outbound_sockets=127.11.40.194
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--builtin_ntp_servers=127.11.40.212:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:28.537741 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.529020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.193:55505 (local address 127.11.40.254:39651)
0929 00:55:28.529174 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.529178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.530346 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 00:55:28.532237 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.532241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.532292 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.532433 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.533760 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.534304 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.535639 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.535887 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.536005 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.537215 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.537219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.537221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.537235 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.537261 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.537270 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.537353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.537515 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.537589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:55:28.538626 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.529431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39651 (local address 127.11.40.193:55505)
0929 00:55:28.530156 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.530199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.532022 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.532621 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.532629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.532767 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:28.533601 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.533624 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.534441 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.534448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.535508 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.535518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.536051 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.536055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.536226 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.536919 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:28.536920 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:28.536961 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:28.537027 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:28.537355 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:28.537367 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:28.537373 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.537503 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.537795 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260929 00:55:28.539903 30764 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39651
I20260929 00:55:28.540177 30764 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:28.540753 30764 heartbeater.cc:511] Master 127.11.40.254:39651 requested a full tablet report, sending...
I20260929 00:55:28.542690 30575 ts_manager.cc:194] Registered new tserver with Master: 2ac6556b87ce40989e90e573dc7acc60 (127.11.40.193:45337)
I20260929 00:55:28.544492 30575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.193:55505
W20260929 00:55:28.653450 30767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.653764 30767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.653805 30767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.657995 30767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.658062 30767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.658119 30767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.194
I20260929 00:55:28.663034 30767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.40.194
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.194
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:28.664197 30767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.665675 30767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:28.675387 30772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.675479 30767 server_base.cc:1034] running on GCE node
W20260929 00:55:28.675417 30773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.675387 30775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.676074 30767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:28.676862 30767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:28.678071 30767 hybrid_clock.cc:648] HybridClock initialized: now 1790643328678041 us; error 48 us; skew 500 ppm
I20260929 00:55:28.680065 30767 webserver.cc:460] Webserver started at http://127.11.40.194:34011/ using document root <none> and password file <none>
I20260929 00:55:28.680644 30767 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:28.680737 30767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:28.681028 30767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:28.683044 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data/instance:
uuid: "6221b22379a54214a7eff8a135f6b487"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "805357decfd892a600d17939516ed1e1"
server_key_iv: "f630589f62244105e3c69ed825bae836"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.683604 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance:
uuid: "6221b22379a54214a7eff8a135f6b487"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "805357decfd892a600d17939516ed1e1"
server_key_iv: "f630589f62244105e3c69ed825bae836"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.687438 30767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:28.690064 30781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.691481 30767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:28.691632 30767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal
uuid: "6221b22379a54214a7eff8a135f6b487"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "805357decfd892a600d17939516ed1e1"
server_key_iv: "f630589f62244105e3c69ed825bae836"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.691766 30767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:28.711894 30767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:28.712627 30767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:28.712949 30767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:28.713630 30767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:28.714701 30767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:28.714773 30767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.714843 30767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:28.714874 30767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.724184 30767 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.194:39945
I20260929 00:55:28.724267 30894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.194:39945 every 8 connection(s)
I20260929 00:55:28.725263 30767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:28.730788 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30767
I20260929 00:55:28.730957 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:28.731251 11427 external_mini_cluster.cc:1428] Setting key aa797df4e5f2b88c2afb53137b44fbcb
I20260929 00:55:28.733515 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.40.195:0
--local_ip_for_outbound_sockets=127.11.40.195
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--builtin_ntp_servers=127.11.40.212:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:55:28.736089 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.727375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.194:43587 (local address 127.11.40.254:39651)
0929 00:55:28.727533 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.727537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.728683 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 00:55:28.730185 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.730188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.730237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.730324 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.731376 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.732076 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.733830 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.734093 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.734200 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.735557 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.735560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.735563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.735584 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.735604 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.735612 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.735696 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.735868 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.735941 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:55:28.737105 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.727786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39651 (local address 127.11.40.194:43587)
0929 00:55:28.728517 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.728561 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.729970 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.730521 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.730528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.730613 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:28.731218 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.731233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.732234 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.732237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.733680 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.733695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.734284 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.734291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.734533 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.735256 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:28.735258 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:28.735303 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:28.735374 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:28.735697 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:28.735709 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:28.735717 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.735859 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.736172 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260929 00:55:28.738363 30895 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39651
I20260929 00:55:28.738632 30895 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:28.739173 30895 heartbeater.cc:511] Master 127.11.40.254:39651 requested a full tablet report, sending...
I20260929 00:55:28.740293 30575 ts_manager.cc:194] Registered new tserver with Master: 6221b22379a54214a7eff8a135f6b487 (127.11.40.194:39945)
I20260929 00:55:28.741281 30575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.194:43587
W20260929 00:55:28.850040 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.850291 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.850332 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.854456 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.854513 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.854569 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.40.195
I20260929 00:55:28.859349 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.40.195
--webserver_port=0
--tserver_master_addrs=127.11.40.254:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.40.195
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:28.860574 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.863054 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:28.874173 30904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.874176 30906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.874287 30898 server_base.cc:1034] running on GCE node
W20260929 00:55:28.874176 30903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.875138 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:28.875941 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:28.877156 30898 hybrid_clock.cc:648] HybridClock initialized: now 1790643328877119 us; error 51 us; skew 500 ppm
I20260929 00:55:28.879352 30898 webserver.cc:460] Webserver started at http://127.11.40.195:39361/ using document root <none> and password file <none>
I20260929 00:55:28.879935 30898 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:28.880054 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:28.880301 30898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:28.882577 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data/instance:
uuid: "401df08fd3f4497fa24337b781d9aa48"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4d234de0986b6c7df2451de4ef20e43e"
server_key_iv: "1996984d35a9a06020ef5e9bd08a006f"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.883183 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance:
uuid: "401df08fd3f4497fa24337b781d9aa48"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4d234de0986b6c7df2451de4ef20e43e"
server_key_iv: "1996984d35a9a06020ef5e9bd08a006f"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.887280 30898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:28.890108 30912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.891676 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:28.891845 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal
uuid: "401df08fd3f4497fa24337b781d9aa48"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-5309"
server_key: "4d234de0986b6c7df2451de4ef20e43e"
server_key_iv: "1996984d35a9a06020ef5e9bd08a006f"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.891985 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:28.916522 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:28.917320 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:28.917537 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:28.918175 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:28.919174 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:28.919251 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.919335 30898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:28.919380 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.928421 30898 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.195:33643
I20260929 00:55:28.928454 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.195:33643 every 8 connection(s)
I20260929 00:55:28.929415 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:28.931320 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 30898
I20260929 00:55:28.931442 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:28.931749 11427 external_mini_cluster.cc:1428] Setting key 670967cab2414657d86f37cec50ace14
I20260929 00:55:28.940147 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.932147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:33979 (local address 127.11.40.254:39651)
0929 00:55:28.932337 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.932341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.933260 (+ 919us) server_negotiation.cc:407] Connection header received
0929 00:55:28.935057 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.935060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.935104 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.935174 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.936143 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.936721 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.938044 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.938320 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.938407 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.939643 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.939646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.939648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.939661 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.939686 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.939694 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.939769 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.939952 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.940011 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:55:28.941116 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.932100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:39651 (local address 127.11.40.195:33979)
0929 00:55:28.933077 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.933141 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.934858 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.935349 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.935356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.935479 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:28.935992 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.936004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.936921 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.936925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.937915 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.937924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.938491 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.938494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.938655 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.939367 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:28.939369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:28.939419 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:28.939488 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:28.939781 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:28.939787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:28.939792 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.939893 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.940126 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":146,"threads_started":1}
I20260929 00:55:28.942405 31026 heartbeater.cc:347] Connected to a master server at 127.11.40.254:39651
I20260929 00:55:28.942657 31026 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:28.943208 31026 heartbeater.cc:511] Master 127.11.40.254:39651 requested a full tablet report, sending...
I20260929 00:55:28.944376 30575 ts_manager.cc:194] Registered new tserver with Master: 401df08fd3f4497fa24337b781d9aa48 (127.11.40.195:33643)
I20260929 00:55:28.945073 30575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.40.195:33979
I20260929 00:55:28.946187 11427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:28.952085 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.948200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.11.40.254:39651)
0929 00:55:28.948329 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.948334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.948536 (+ 202us) server_negotiation.cc:407] Connection header received
0929 00:55:28.948629 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.948632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.948669 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.948737 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.949019 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.949537 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.950434 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.950644 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.950727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.951559 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.951562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.951564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.951575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.951597 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.951605 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.951687 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.951907 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.951970 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:55:28.955539 30575 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:28.958714 30575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:28.975567 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.969379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.11.40.194:39945)
0929 00:55:28.970613 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.970627 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.970658 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:55:28.970742 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.970751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.971005 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.971192 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:28.971535 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.972400 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.973962 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.974807 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.975082 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.975296 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.975352 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.975432 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":99,"threads_started":1}
I20260929 00:55:28.975577 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.969315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39945 (local address 127.0.0.1:38112)
0929 00:55:28.970497 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.970512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.970600 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.971187 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.971195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.971228 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:28.971394 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.971402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.972527 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.972532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.973770 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.973780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.974943 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.974946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.975140 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.975182 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.975293 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.975423 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":47,"thread_start_us":138,"threads_started":1}
I20260929 00:55:28.976374 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.969301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.195:33643 (local address 127.0.0.1:50258)
0929 00:55:28.969911 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.969971 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.970122 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.971390 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.971396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.971418 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:28.971530 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.971537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.973099 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.973102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.974115 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.974124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.974945 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.974948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.976040 (+ 1092us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.976095 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.976168 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.976238 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":44,"thread_start_us":89,"threads_started":1}
I20260929 00:55:28.976850 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.969515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45337 (local address 127.0.0.1:37488)
0929 00:55:28.970619 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:28.970636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:28.970722 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:28.971547 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:28.971551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:28.971568 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:28.971649 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.971654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.973934 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.973940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.975175 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:28.975185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.976438 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.976441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:28.976534 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.976556 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:55:28.976626 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 00:55:28.976686 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":75,"thread_start_us":94,"threads_started":1}
I20260929 00:55:28.976861 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.969935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.11.40.195:33643)
0929 00:55:28.970882 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.970894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.970920 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:28.970997 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.971002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.971214 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.971375 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:28.971655 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.972924 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.974243 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.974799 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.974946 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.975179 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.976557 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.976671 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":161,"threads_started":1}
I20260929 00:55:28.977295 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.970137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.11.40.193:45337)
0929 00:55:28.970998 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.971010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.971034 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:55:28.971111 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.971119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.971301 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.971448 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:28.971774 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.972939 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.975285 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.976088 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.976191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.976358 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.977088 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.977174 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":147,"threads_started":1}
I20260929 00:55:28.979389 30960 tablet_service.cc:1467] Processing CreateTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e (DEFAULT_TABLE table=test-table [id=cdd3190a6cf24716b5f54a95dbd58f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:28.979275 30829 tablet_service.cc:1467] Processing CreateTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e (DEFAULT_TABLE table=test-table [id=cdd3190a6cf24716b5f54a95dbd58f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:28.980414 30960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90989193f8e54d039d8f4ab15c8edf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:28.980024 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e (DEFAULT_TABLE table=test-table [id=cdd3190a6cf24716b5f54a95dbd58f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:28.980634 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90989193f8e54d039d8f4ab15c8edf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:28.981261 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90989193f8e54d039d8f4ab15c8edf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:28.986999 31043 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Bootstrap starting.
I20260929 00:55:28.988481 31045 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Bootstrap starting.
I20260929 00:55:28.989127 31044 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Bootstrap starting.
I20260929 00:55:28.990213 31043 tablet_bootstrap.cc:654] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:28.990748 31045 tablet_bootstrap.cc:654] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:28.991164 31043 log.cc:824] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:28.991325 31044 tablet_bootstrap.cc:654] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:28.991590 31045 log.cc:824] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:28.992172 31044 log.cc:824] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:28.993340 31043 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: No bootstrap required, opened a new log
I20260929 00:55:28.993570 31043 ts_tablet_manager.cc:1397] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 00:55:28.994171 31044 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: No bootstrap required, opened a new log
I20260929 00:55:28.994372 31044 ts_tablet_manager.cc:1397] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 00:55:28.997599 31043 raft_consensus.cc:348] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:28.997879 31043 raft_consensus.cc:374] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:28.997941 31043 raft_consensus.cc:729] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401df08fd3f4497fa24337b781d9aa48, State: Initialized, Role: FOLLOWER
I20260929 00:55:28.998436 31043 consensus_queue.cc:260] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:28.998744 31044 raft_consensus.cc:348] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:28.999053 31044 raft_consensus.cc:374] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:28.999121 31044 raft_consensus.cc:729] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6221b22379a54214a7eff8a135f6b487, State: Initialized, Role: FOLLOWER
I20260929 00:55:28.999357 31026 heartbeater.cc:503] Master 127.11.40.254:39651 was elected leader, sending a full tablet report...
I20260929 00:55:28.999581 31043 ts_tablet_manager.cc:1428] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 00:55:28.999751 31044 consensus_queue.cc:260] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.001091 30895 heartbeater.cc:503] Master 127.11.40.254:39651 was elected leader, sending a full tablet report...
I20260929 00:55:29.001755 31044 ts_tablet_manager.cc:1428] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260929 00:55:29.001978 31045 tablet_bootstrap.cc:492] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: No bootstrap required, opened a new log
I20260929 00:55:29.002189 31045 ts_tablet_manager.cc:1397] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260929 00:55:29.006625 31045 raft_consensus.cc:348] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.006932 31045 raft_consensus.cc:374] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:29.006971 31045 raft_consensus.cc:729] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac6556b87ce40989e90e573dc7acc60, State: Initialized, Role: FOLLOWER
I20260929 00:55:29.007381 31045 consensus_queue.cc:260] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.008176 30764 heartbeater.cc:503] Master 127.11.40.254:39651 was elected leader, sending a full tablet report...
I20260929 00:55:29.008663 31045 ts_tablet_manager.cc:1428] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 00:55:29.028290 30765 tablet.cc:2369] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:55:29.084610 31049 raft_consensus.cc:482] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:29.084924 31049 raft_consensus.cc:504] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.086153 31049 leader_election.cc:288] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6221b22379a54214a7eff8a135f6b487 (127.11.40.194:39945), 2ac6556b87ce40989e90e573dc7acc60 (127.11.40.193:45337)
I20260929 00:55:29.091274 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.086575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:50921 (local address 127.11.40.193:45337)
0929 00:55:29.086727 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.086731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.086749 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:55:29.086818 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.086820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.086883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.086953 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:29.087271 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.088091 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.089854 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.090528 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.090701 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.090770 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.091031 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.091111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:55:29.091311 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.086673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.40.195:44461 (local address 127.11.40.194:39945)
0929 00:55:29.086855 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.086860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.087042 (+ 182us) server_negotiation.cc:407] Connection header received
0929 00:55:29.087136 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.087138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.087211 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.087290 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:29.087582 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.088371 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.089854 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.090602 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.090720 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.090770 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.091031 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.091111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 00:55:29.091280 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.086454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45337 (local address 127.11.40.195:50921)
0929 00:55:29.086574 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.086592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:29.086695 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:29.087001 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:29.087006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:29.087028 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:29.087173 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.087179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.088242 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.088245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.089643 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.089656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.090706 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.090710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.090873 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.090913 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:55:29.090995 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:55:29.091110 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":112}
I20260929 00:55:29.091288 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.086515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.194:39945 (local address 127.11.40.195:44461)
0929 00:55:29.086907 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.086925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:29.087044 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:29.087321 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:29.087323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:29.087338 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:29.087455 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.087463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.088499 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.088502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.089709 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.089717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.090711 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.090714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.090866 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.090910 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 00:55:29.090995 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 00:55:29.091110 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":43,"spinlock_wait_cycles":3840,"thread_start_us":137,"threads_started":1}
I20260929 00:55:29.092114 30849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90989193f8e54d039d8f4ab15c8edf7e" candidate_uuid: "401df08fd3f4497fa24337b781d9aa48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6221b22379a54214a7eff8a135f6b487" is_pre_election: true
I20260929 00:55:29.092108 30718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90989193f8e54d039d8f4ab15c8edf7e" candidate_uuid: "401df08fd3f4497fa24337b781d9aa48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac6556b87ce40989e90e573dc7acc60" is_pre_election: true
I20260929 00:55:29.092447 30718 raft_consensus.cc:2454] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401df08fd3f4497fa24337b781d9aa48 in term 0.
I20260929 00:55:29.092447 30849 raft_consensus.cc:2454] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401df08fd3f4497fa24337b781d9aa48 in term 0.
I20260929 00:55:29.092976 30914 leader_election.cc:302] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ac6556b87ce40989e90e573dc7acc60, 401df08fd3f4497fa24337b781d9aa48, 6221b22379a54214a7eff8a135f6b487; no voters:
I20260929 00:55:29.093261 31049 raft_consensus.cc:2781] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:29.093349 31049 raft_consensus.cc:482] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:29.093393 31049 raft_consensus.cc:3037] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:29.094393 31049 raft_consensus.cc:504] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.094751 31049 leader_election.cc:288] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [CANDIDATE]: Term 1 election: Requested vote from peers 6221b22379a54214a7eff8a135f6b487 (127.11.40.194:39945), 2ac6556b87ce40989e90e573dc7acc60 (127.11.40.193:45337)
I20260929 00:55:29.095052 30849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90989193f8e54d039d8f4ab15c8edf7e" candidate_uuid: "401df08fd3f4497fa24337b781d9aa48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6221b22379a54214a7eff8a135f6b487"
I20260929 00:55:29.095180 30718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90989193f8e54d039d8f4ab15c8edf7e" candidate_uuid: "401df08fd3f4497fa24337b781d9aa48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac6556b87ce40989e90e573dc7acc60"
I20260929 00:55:29.095193 30849 raft_consensus.cc:3037] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:29.095316 30718 raft_consensus.cc:3037] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:29.096365 30849 raft_consensus.cc:2454] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401df08fd3f4497fa24337b781d9aa48 in term 1.
I20260929 00:55:29.096640 30718 raft_consensus.cc:2454] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401df08fd3f4497fa24337b781d9aa48 in term 1.
I20260929 00:55:29.097076 30914 leader_election.cc:302] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401df08fd3f4497fa24337b781d9aa48, 6221b22379a54214a7eff8a135f6b487; no voters:
I20260929 00:55:29.097359 31049 raft_consensus.cc:2781] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:29.097600 31049 raft_consensus.cc:686] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 1 LEADER]: Becoming Leader. State: Replica: 401df08fd3f4497fa24337b781d9aa48, State: Running, Role: LEADER
I20260929 00:55:29.097945 31049 consensus_queue.cc:237] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } }
I20260929 00:55:29.101532 30575 catalog_manager.cc:5495] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401df08fd3f4497fa24337b781d9aa48 (127.11.40.195). New cstate: current_term: 1 leader_uuid: "401df08fd3f4497fa24337b781d9aa48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401df08fd3f4497fa24337b781d9aa48" member_type: VOTER last_known_addr { host: "127.11.40.195" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:29.141163 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.137191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.11.40.195:33643)
0929 00:55:29.137341 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.137347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.137365 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:55:29.137476 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.137478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.137535 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.137662 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:29.137966 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.138533 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.139504 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.139754 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.139880 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.140228 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:29.140681 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:29.140853 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.140923 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.140978 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:55:29.147382 30849 raft_consensus.cc:1261] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 1 FOLLOWER]: Refusing update from remote peer 401df08fd3f4497fa24337b781d9aa48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:29.147362 30718 raft_consensus.cc:1261] T 90989193f8e54d039d8f4ab15c8edf7e P 2ac6556b87ce40989e90e573dc7acc60 [term 1 FOLLOWER]: Refusing update from remote peer 401df08fd3f4497fa24337b781d9aa48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:29.148137 31053 consensus_queue.cc:1035] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac6556b87ce40989e90e573dc7acc60" member_type: VOTER last_known_addr { host: "127.11.40.193" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:29.148356 31049 consensus_queue.cc:1035] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6221b22379a54214a7eff8a135f6b487" member_type: VOTER last_known_addr { host: "127.11.40.194" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:29.158514 31057 mvcc.cc:205] Tried to move back new op lower bound from 7334475076179730432 to 7334475075987701760. Current Snapshot: MvccSnapshot[applied={T|T < 7334475076179730432}]
I20260929 00:55:29.165468 30574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:29.165680 30574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:29.168606 30574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5140b52880a54412ab08d3798d63316a: Sending DeleteTablet for 3 replicas of tablet 90989193f8e54d039d8f4ab15c8edf7e
I20260929 00:55:29.169549 30829 tablet_service.cc:1514] Processing DeleteTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:29 UTC) from {username='slave'} at 127.0.0.1:38112
I20260929 00:55:29.169677 30960 tablet_service.cc:1514] Processing DeleteTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:29 UTC) from {username='slave'} at 127.0.0.1:50258
I20260929 00:55:29.170162 31066 tablet_replica.cc:330] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: stopping tablet replica
I20260929 00:55:29.170223 31067 tablet_replica.cc:330] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: stopping tablet replica
I20260929 00:55:29.170428 30698 tablet_service.cc:1514] Processing DeleteTablet for tablet 90989193f8e54d039d8f4ab15c8edf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:29 UTC) from {username='slave'} at 127.0.0.1:37488
I20260929 00:55:29.170574 31067 raft_consensus.cc:2229] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:29.170681 31066 raft_consensus.cc:2229] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:29.171011 31067 raft_consensus.cc:2258] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:29.171051 31066 raft_consensus.cc:2258] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:29.171288 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30635
I20260929 00:55:29.172963 31066 ts_tablet_manager.cc:1905] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:29.173174 31067 ts_tablet_manager.cc:1905] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:29.176142 31066 ts_tablet_manager.cc:1918] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:29.176142 31067 ts_tablet_manager.cc:1918] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:29.176258 31067 log.cc:1196] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-2/wal/wals/90989193f8e54d039d8f4ab15c8edf7e
I20260929 00:55:29.176407 31066 log.cc:1196] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Deleting WAL directory at /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790642980693194-11427-0/minicluster-data/ts-1/wal/wals/90989193f8e54d039d8f4ab15c8edf7e
I20260929 00:55:29.176618 31067 ts_tablet_manager.cc:1939] T 90989193f8e54d039d8f4ab15c8edf7e P 401df08fd3f4497fa24337b781d9aa48: Deleting consensus metadata
I20260929 00:55:29.176939 31066 ts_tablet_manager.cc:1939] T 90989193f8e54d039d8f4ab15c8edf7e P 6221b22379a54214a7eff8a135f6b487: Deleting consensus metadata
I20260929 00:55:29.178380 30561 catalog_manager.cc:4841] TS 401df08fd3f4497fa24337b781d9aa48 (127.11.40.195:33643): tablet 90989193f8e54d039d8f4ab15c8edf7e (table test-table [id=cdd3190a6cf24716b5f54a95dbd58f80]) successfully deleted
I20260929 00:55:29.178380 30560 catalog_manager.cc:4841] TS 6221b22379a54214a7eff8a135f6b487 (127.11.40.194:39945): tablet 90989193f8e54d039d8f4ab15c8edf7e (table test-table [id=cdd3190a6cf24716b5f54a95dbd58f80]) successfully deleted
W20260929 00:55:29.181813 30559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.40.193:45337 (error 108)
I20260929 00:55:29.182683 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30767
I20260929 00:55:29.183056 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.182697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.193:45337 (local address 127.0.0.1:37504)
0929 00:55:29.182863 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.182946 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 00:55:29.183346 30559 catalog_manager.cc:4584] TS 2ac6556b87ce40989e90e573dc7acc60 (127.11.40.193:45337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90989193f8e54d039d8f4ab15c8edf7e: Network error: Client connection negotiation failed: client connection to 127.11.40.193:45337: connect: Connection refused (error 111)
I20260929 00:55:29.189976 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30898
I20260929 00:55:29.196264 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 30544
2026-09-29T00:55:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1130 ms)
[----------] 6 tests from EncryptionPolicyTest (10739 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:29 dist-test-slave-5309 krb5kdc[31072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:29 dist-test-slave-5309 krb5kdc[31072](info): set up 2 sockets
Sep 29 00:55:29 dist-test-slave-5309 krb5kdc[31072](info): commencing operation
krb5kdc: starting...
W20260929 00:55:30.270725 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:30.405109 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:30.527746 31085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:30.528072 31085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:30.528127 31085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:30.532444 31085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:30.532516 31085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:30.532536 31085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:30.532552 31085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:30.532568 31085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:30.538756 31085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:30.540022 31085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:30.541123 31085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:30.550530 31093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:30.550540 31090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:30.550540 31091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:30.550638 31085 server_base.cc:1034] running on GCE node
I20260929 00:55:30.551456 31085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:30.552721 31085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:30.553929 31085 hybrid_clock.cc:648] HybridClock initialized: now 1790643330553913 us; error 30 us; skew 500 ppm
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:30.565951 31085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:30.567163 31085 webserver.cc:460] Webserver started at http://127.0.0.1:41721/ using document root <none> and password file <none>
I20260929 00:55:30.568102 31085 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:30.568216 31085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:30.568485 31085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:30.570819 31085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "78ac2c67587242dd8ceadd121e616168"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-5309"
server_key: "e6da7a5591794113d45f71a2adb3252f"
server_key_iv: "261f8191ef061d018b317a20205f4555"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.571429 31085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "78ac2c67587242dd8ceadd121e616168"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-5309"
server_key: "e6da7a5591794113d45f71a2adb3252f"
server_key_iv: "261f8191ef061d018b317a20205f4555"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.575564 31085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:30.578162 31100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:30.579522 31085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:55:30.579715 31085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "78ac2c67587242dd8ceadd121e616168"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-5309"
server_key: "e6da7a5591794113d45f71a2adb3252f"
server_key_iv: "261f8191ef061d018b317a20205f4555"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.579852 31085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:30.598143 31085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:30.602442 31085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:30.602698 31085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:30.611558 31085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37559
I20260929 00:55:30.611615 31152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37559 every 8 connection(s)
I20260929 00:55:30.612673 31085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:30.613189 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31085
I20260929 00:55:30.613353 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:30.613786 11427 external_mini_cluster.cc:1428] Setting key ccf0507fbb536b39fe755b8887990f05
I20260929 00:55:30.616291 31153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:30.622540 31153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168: Bootstrap starting.
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:55:30.626163 31153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:30.627130 31153 log.cc:824] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:30.630025 31153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168: No bootstrap required, opened a new log
I20260929 00:55:30.630681 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.615441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53415 (local address 127.0.0.1:37559)
0929 00:55:30.615943 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.615960 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.615998 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:30.616789 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.616868 (+ 79us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.617238 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.617607 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.618482 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.619539 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.620431 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.621002 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.621182 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:30.624991 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.624997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.625019 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:30.625059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.627064 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:30.627857 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:30.627864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:30.627870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:30.628044 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:30.628486 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:30.628489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:30.628490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:30.628987 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.629141 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.629532 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.629787 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":144,"threads_started":1}
I20260929 00:55:30.633946 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:30.634382 31153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } }
I20260929 00:55:30.634757 31153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:30.634837 31153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ac2c67587242dd8ceadd121e616168, State: Initialized, Role: FOLLOWER
I20260929 00:55:30.635417 31153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } }
I20260929 00:55:30.635637 31153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:30.635722 31153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:30.635836 31153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:30.637393 31153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } }
I20260929 00:55:30.637869 31153 leader_election.cc:302] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78ac2c67587242dd8ceadd121e616168; no voters:
I20260929 00:55:30.638255 31153 leader_election.cc:288] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:30.638372 31163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:30.638628 31163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [term 1 LEADER]: Becoming Leader. State: Replica: 78ac2c67587242dd8ceadd121e616168, State: Running, Role: LEADER
I20260929 00:55:30.639035 31163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } }
I20260929 00:55:30.639858 31153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:30.640277 31163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78ac2c67587242dd8ceadd121e616168. Latest consensus state: current_term: 1 leader_uuid: "78ac2c67587242dd8ceadd121e616168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } } }
I20260929 00:55:30.641108 31163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:30.640983 31164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78ac2c67587242dd8ceadd121e616168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78ac2c67587242dd8ceadd121e616168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37559 } } }
I20260929 00:55:30.641489 31164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:30.643270 31172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:30.646912 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.636540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36291 (local address 127.0.0.1:37559)
0929 00:55:30.636690 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.636694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.636941 (+ 247us) server_negotiation.cc:407] Connection header received
0929 00:55:30.637091 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.637093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.637135 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.637205 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.638111 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.638924 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.640132 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.640480 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.640589 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:30.642765 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.642768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.642771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:30.642796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.644379 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:30.644941 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:30.644945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:30.644947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:30.645038 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:30.645391 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:30.645394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:30.645396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:30.645593 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.645753 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.646574 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.646703 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:30.646922 31172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:30.653934 31172 catalog_manager.cc:1347] Generated new cluster ID: 9fd8465e015d40239c6633410c12468e
I20260929 00:55:30.654050 31172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:30.666844 31172 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:30.667939 31172 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:30.672489 31172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78ac2c67587242dd8ceadd121e616168: Generated new TSK 0
I20260929 00:55:30.673439 31172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:30.678787 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31085
W20260929 00:55:30.684706 31070 connection.cc:535] client connection to 127.0.0.1:37559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37559): Connection reset by peer (error 104)
2026-09-29T00:55:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31184](info): set up 2 sockets
Sep 29 00:55:30 dist-test-slave-5309 krb5kdc[31184](info): commencing operation
krb5kdc: starting...
W20260929 00:55:31.742383 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:31 dist-test-slave-5309 krb5kdc[31184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:31.885071 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:32.002555 31197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:32.002969 31197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:32.003073 31197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:32.007550 31197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:32.007685 31197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:32.007726 31197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:32.007768 31197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:32.007812 31197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:32.013844 31197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:32.015395 31197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:32.016570 31197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:32.026451 31202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:32.027068 31205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:32.027076 31203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:32.027319 31197 server_base.cc:1034] running on GCE node
I20260929 00:55:32.028095 31197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:32.029495 31197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:32.030697 31197 hybrid_clock.cc:648] HybridClock initialized: now 1790643332030682 us; error 31 us; skew 500 ppm
Sep 29 00:55:32 dist-test-slave-5309 krb5kdc[31184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:32.033877 31197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:32.034950 31197 webserver.cc:460] Webserver started at http://127.0.0.1:44105/ using document root <none> and password file <none>
I20260929 00:55:32.035545 31197 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:32.035609 31197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:32.035830 31197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:32.038120 31197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "c96bf2a73dc34197918e97dcda8efed0"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-5309"
server_key: "bc70949621063b1cef8edfd2d8fd1994"
server_key_iv: "faaa93baeb4bedbbe7250f7c5f259f2b"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.038777 31197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "c96bf2a73dc34197918e97dcda8efed0"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-5309"
server_key: "bc70949621063b1cef8edfd2d8fd1994"
server_key_iv: "faaa93baeb4bedbbe7250f7c5f259f2b"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.043166 31197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:32.046135 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:32.047612 31197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:32.047736 31197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "c96bf2a73dc34197918e97dcda8efed0"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-5309"
server_key: "bc70949621063b1cef8edfd2d8fd1994"
server_key_iv: "faaa93baeb4bedbbe7250f7c5f259f2b"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.047914 31197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:32.073465 31197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:32.077628 31197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:32.077842 31197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:32.086790 31197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34041
I20260929 00:55:32.086799 31264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34041 every 8 connection(s)
I20260929 00:55:32.088007 31197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:32.090978 31265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:32.092693 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31197
I20260929 00:55:32.092866 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:32.093182 11427 external_mini_cluster.cc:1428] Setting key 965abebc0b2c1136c5a4f5f8f2d733be
I20260929 00:55:32.098078 31265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0: Bootstrap starting.
I20260929 00:55:32.101055 31265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:32.101897 31265 log.cc:824] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:32.103282 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.094684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45019 (local address 127.0.0.1:34041)
0929 00:55:32.095423 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.095445 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.095494 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:55:32.096180 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.096198 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.096571 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.096953 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.097310 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:32.098433 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:32.099691 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:32.100213 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:32.100346 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:32.101419 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.101425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.101460 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.101872 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.101933 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.101957 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.102122 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.102324 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.102489 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260929 00:55:32.104110 31265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0: No bootstrap required, opened a new log
I20260929 00:55:32.106719 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:32.108712 31265 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } }
I20260929 00:55:32.109071 31265 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:32.109158 31265 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96bf2a73dc34197918e97dcda8efed0, State: Initialized, Role: FOLLOWER
I20260929 00:55:32.109774 31265 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } }
I20260929 00:55:32.109972 31265 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:32.110050 31265 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:32.110175 31265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:32.111457 31265 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } }
I20260929 00:55:32.111819 31265 leader_election.cc:302] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96bf2a73dc34197918e97dcda8efed0; no voters:
I20260929 00:55:32.112175 31265 leader_election.cc:288] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:32.112331 31275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:32.112562 31275 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [term 1 LEADER]: Becoming Leader. State: Replica: c96bf2a73dc34197918e97dcda8efed0, State: Running, Role: LEADER
I20260929 00:55:32.113276 31275 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } }
I20260929 00:55:32.113775 31265 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:32.114164 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.109358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57967 (local address 127.0.0.1:34041)
0929 00:55:32.109492 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.109496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.109617 (+ 121us) server_negotiation.cc:407] Connection header received
0929 00:55:32.109719 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.109722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.109771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.109867 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.110236 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:32.111000 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:32.111943 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:32.112347 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:32.112912 (+ 565us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:32.113472 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.113476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.113479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.113492 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.113520 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.113532 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.113915 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.114001 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.114038 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:32.115097 31276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96bf2a73dc34197918e97dcda8efed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } } }
I20260929 00:55:32.115234 31276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:32.115996 31284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:32.119235 31284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:32.120970 31277 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96bf2a73dc34197918e97dcda8efed0. Latest consensus state: current_term: 1 leader_uuid: "c96bf2a73dc34197918e97dcda8efed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bf2a73dc34197918e97dcda8efed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34041 } } }
I20260929 00:55:32.121104 31277 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:32.125862 31284 catalog_manager.cc:1347] Generated new cluster ID: e84a94115cb24f9da1488e8794b51693
I20260929 00:55:32.125949 31284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:32.157526 31284 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:32.158831 31284 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:32.174765 31284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96bf2a73dc34197918e97dcda8efed0: Generated new TSK 0
I20260929 00:55:32.175678 31284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:32.190202 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31197
W20260929 00:55:32.196112 31182 connection.cc:535] client connection to 127.0.0.1:34041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34041): Connection reset by peer (error 104)
2026-09-29T00:55:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:32 dist-test-slave-5309 krb5kdc[31296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:32 dist-test-slave-5309 krb5kdc[31296](info): set up 2 sockets
Sep 29 00:55:32 dist-test-slave-5309 krb5kdc[31296](info): commencing operation
krb5kdc: starting...
W20260929 00:55:33.310587 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:33 dist-test-slave-5309 krb5kdc[31296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:33.446091 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:33.559218 31309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:33.559590 31309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:33.559675 31309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:33.563933 31309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:33.564041 31309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:33.564078 31309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:33.564121 31309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:33.564153 31309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:33.569695 31309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:33.570892 31309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:33.571995 31309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:33.581599 31317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:33.581599 31315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:33.581604 31314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:33.581724 31309 server_base.cc:1034] running on GCE node
I20260929 00:55:33.582554 31309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:33.583817 31309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:33.585029 31309 hybrid_clock.cc:648] HybridClock initialized: now 1790643333584991 us; error 55 us; skew 500 ppm
Sep 29 00:55:33 dist-test-slave-5309 krb5kdc[31296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:33.587886 31309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:33.588843 31309 webserver.cc:460] Webserver started at http://127.0.0.1:37701/ using document root <none> and password file <none>
I20260929 00:55:33.589422 31309 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:33.589516 31309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:33.589766 31309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:33.591868 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "dcedafe7c7f94af38bf788e02e57c8ab"
format_stamp: "Formatted at 2026-09-29 00:55:33 on dist-test-slave-5309"
server_key: "4c7643636ed01f9579f4765d0ea6a9b6"
server_key_iv: "345ec80d4ab70e325dfe30105036aab7"
server_key_version: "encryptionkey@0"
I20260929 00:55:33.592455 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "dcedafe7c7f94af38bf788e02e57c8ab"
format_stamp: "Formatted at 2026-09-29 00:55:33 on dist-test-slave-5309"
server_key: "4c7643636ed01f9579f4765d0ea6a9b6"
server_key_iv: "345ec80d4ab70e325dfe30105036aab7"
server_key_version: "encryptionkey@0"
I20260929 00:55:33.596297 31309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:33.598898 31324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:33.600621 31309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:55:33.600780 31309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "dcedafe7c7f94af38bf788e02e57c8ab"
format_stamp: "Formatted at 2026-09-29 00:55:33 on dist-test-slave-5309"
server_key: "4c7643636ed01f9579f4765d0ea6a9b6"
server_key_iv: "345ec80d4ab70e325dfe30105036aab7"
server_key_version: "encryptionkey@0"
I20260929 00:55:33.600944 31309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:33.606278 31309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:33.609704 31309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:33.609949 31309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:33.617660 31309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33133
I20260929 00:55:33.617692 31376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33133 every 8 connection(s)
I20260929 00:55:33.618755 31309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:33.622007 31377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:33.623003 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31309
I20260929 00:55:33.623167 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:33.623588 11427 external_mini_cluster.cc:1428] Setting key 665c694944fa35bf53de5c77248c839c
I20260929 00:55:33.629334 31377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab: Bootstrap starting.
I20260929 00:55:33.629725 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:33.625200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41931 (local address 127.0.0.1:33133)
0929 00:55:33.625750 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:33.625767 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:33.625809 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:55:33.626514 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:33.626532 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:33.626859 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:33.627270 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:33.628249 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:33.628256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:33.628273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:33.628286 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:33.628351 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:33.628373 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:33.628542 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:33.628621 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 00:55:33.628875 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":158,"threads_started":1}
I20260929 00:55:33.632130 31377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:33.632889 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:33.633144 31377 log.cc:824] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:33.635286 31377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab: No bootstrap required, opened a new log
I20260929 00:55:33.636885 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:33.635096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39319 (local address 127.0.0.1:33133)
0929 00:55:33.635229 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:33.635235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:33.635495 (+ 260us) server_negotiation.cc:407] Connection header received
0929 00:55:33.635593 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:33.635596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:33.635633 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:33.635739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:33.636434 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:33.636438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:33.636441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:33.636454 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:33.636486 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:33.636502 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:33.636601 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:33.636668 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 00:55:33.636710 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:55:33.639897 31377 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260929 00:55:33.640146 31377 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:33.640214 31377 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcedafe7c7f94af38bf788e02e57c8ab, State: Initialized, Role: FOLLOWER
I20260929 00:55:33.640748 31377 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260929 00:55:33.640985 31377 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:33.641062 31377 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:33.641173 31377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:33.642244 31377 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260929 00:55:33.642640 31377 leader_election.cc:302] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcedafe7c7f94af38bf788e02e57c8ab; no voters:
I20260929 00:55:33.643016 31377 leader_election.cc:288] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:33.643143 31387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:33.643751 31387 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [term 1 LEADER]: Becoming Leader. State: Replica: dcedafe7c7f94af38bf788e02e57c8ab, State: Running, Role: LEADER
I20260929 00:55:33.644178 31387 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260929 00:55:33.644523 31377 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:33.646130 31388 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } } }
I20260929 00:55:33.646278 31388 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:33.646660 31389 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcedafe7c7f94af38bf788e02e57c8ab. Latest consensus state: current_term: 1 leader_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcedafe7c7f94af38bf788e02e57c8ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } } }
I20260929 00:55:33.646744 31396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:33.646857 31389 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:33.650128 31396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:33.656096 31396 catalog_manager.cc:1347] Generated new cluster ID: 9ec4549e5ca74dfba652327def708122
I20260929 00:55:33.656255 31396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:33.676126 31396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:33.676503 31396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:33.686498 31396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcedafe7c7f94af38bf788e02e57c8ab: Generated new TSK 0
I20260929 00:55:33.687408 31396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:33.695399 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31309
2026-09-29T00:55:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:33 dist-test-slave-5309 krb5kdc[31408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:33 dist-test-slave-5309 krb5kdc[31408](info): set up 2 sockets
Sep 29 00:55:33 dist-test-slave-5309 krb5kdc[31408](info): commencing operation
krb5kdc: starting...
W20260929 00:55:34.778481 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:34 dist-test-slave-5309 krb5kdc[31408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:34.912665 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:35.030336 31421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:35.030700 31421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:35.030748 31421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:35.037742 31421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:35.037835 31421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:35.037856 31421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:35.037875 31421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:35.037897 31421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:35.043876 31421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:35.045140 31421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:35.046176 31421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:35.055706 31427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.055649 31429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.055658 31426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:35.056102 31421 server_base.cc:1034] running on GCE node
I20260929 00:55:35.056911 31421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:35.058187 31421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:35.059391 31421 hybrid_clock.cc:648] HybridClock initialized: now 1790643335059369 us; error 35 us; skew 500 ppm
Sep 29 00:55:35 dist-test-slave-5309 krb5kdc[31408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:35.062695 31421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:35.063730 31421 webserver.cc:460] Webserver started at http://127.0.0.1:39997/ using document root <none> and password file <none>
I20260929 00:55:35.064410 31421 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:35.064481 31421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:35.064710 31421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:35.067459 31421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "6bd2d96e1afa4cf88def8f66e0cbd000"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-5309"
server_key: "d2c76a92297c56d9e0fea1578555ff76"
server_key_iv: "74c6ce89d840aadfd3b1bf12fa28b6e8"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.068114 31421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "6bd2d96e1afa4cf88def8f66e0cbd000"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-5309"
server_key: "d2c76a92297c56d9e0fea1578555ff76"
server_key_iv: "74c6ce89d840aadfd3b1bf12fa28b6e8"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.072960 31421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 00:55:35.076020 31436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.077622 31421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:35.077803 31421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "6bd2d96e1afa4cf88def8f66e0cbd000"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-5309"
server_key: "d2c76a92297c56d9e0fea1578555ff76"
server_key_iv: "74c6ce89d840aadfd3b1bf12fa28b6e8"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.077957 31421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:35.102193 31421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:35.105906 31421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:35.106148 31421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:35.114137 31421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37505
I20260929 00:55:35.114166 31488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37505 every 8 connection(s)
I20260929 00:55:35.115218 31421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:35.118310 31489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:35.120273 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31421
I20260929 00:55:35.120407 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:35.120762 11427 external_mini_cluster.cc:1428] Setting key f8ed40b803567cf3cad48b7daf7fd55c
I20260929 00:55:35.125998 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000: Bootstrap starting.
I20260929 00:55:35.129082 31489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:35.130019 31489 log.cc:824] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:55:35 dist-test-slave-5309 krb5kdc[31408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:55:35.132423 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000: No bootstrap required, opened a new log
I20260929 00:55:35.137236 31489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260929 00:55:35.137600 31489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:35.137717 31489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2d96e1afa4cf88def8f66e0cbd000, State: Initialized, Role: FOLLOWER
I20260929 00:55:35.138321 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.122396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58033 (local address 127.0.0.1:37505)
0929 00:55:35.123117 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:35.123134 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:35.123172 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:35.123848 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:35.123866 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:35.124244 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:35.124550 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:35.125609 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.127186 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.128241 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.129000 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.129199 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.132667 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:35.132674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:35.132690 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:35.132726 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:35.134989 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.135548 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.135554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.135559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.135681 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.135996 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.136000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.136002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.136441 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:35.136704 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:35.137047 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 00:55:35.137372 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":369,"threads_started":1}
I20260929 00:55:35.138281 31489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260929 00:55:35.138456 31489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:35.138537 31489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:35.138652 31489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:35.140033 31489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260929 00:55:35.140460 31489 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd2d96e1afa4cf88def8f66e0cbd000; no voters:
I20260929 00:55:35.140868 31489 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:35.141120 31494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:35.141377 31494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd2d96e1afa4cf88def8f66e0cbd000, State: Running, Role: LEADER
I20260929 00:55:35.141791 31494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260929 00:55:35.141955 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:35.142668 31489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:35.143318 31494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd2d96e1afa4cf88def8f66e0cbd000. Latest consensus state: current_term: 1 leader_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260929 00:55:35.143532 31494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:35.143762 31496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd2d96e1afa4cf88def8f66e0cbd000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260929 00:55:35.143862 31496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:35.144116 31501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:35.147764 31501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:35.156301 31501 catalog_manager.cc:1347] Generated new cluster ID: 8cca205990ab4dcba55d66a4078299e7
I20260929 00:55:35.156399 31501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:35.159276 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.145359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41841 (local address 127.0.0.1:37505)
0929 00:55:35.145485 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:35.145490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:35.148511 (+ 3021us) server_negotiation.cc:407] Connection header received
0929 00:55:35.148700 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:35.148703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:35.148756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:35.148896 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:35.149817 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.150956 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.152586 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.152993 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.153108 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.155228 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:35.155232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:35.155235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:35.155269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:35.157427 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.157947 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.157952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.157954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.158045 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.158348 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.158352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.158354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.158554 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:35.158691 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:35.158963 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 00:55:35.159088 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 00:55:35.182533 31501 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:35.183830 31501 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:35.190585 31501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd2d96e1afa4cf88def8f66e0cbd000: Generated new TSK 0
I20260929 00:55:35.191347 31501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:35.202744 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31421
2026-09-29T00:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:35 dist-test-slave-5309 krb5kdc[31520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:35 dist-test-slave-5309 krb5kdc[31520](info): set up 2 sockets
Sep 29 00:55:35 dist-test-slave-5309 krb5kdc[31520](info): commencing operation
krb5kdc: starting...
W20260929 00:55:36.262542 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:36 dist-test-slave-5309 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:36.392519 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:36.506903 31533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:36.507205 31533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:36.507248 31533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:36.511606 31533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:36.511680 31533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:36.511701 31533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:36.511718 31533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:36.511734 31533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:36.517234 31533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:36.518322 31533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:36.519301 31533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:36.528370 31538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:36.528383 31539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:36.528503 31541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:36.528548 31533 server_base.cc:1034] running on GCE node
I20260929 00:55:36.529419 31533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:36.530668 31533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:36.531890 31533 hybrid_clock.cc:648] HybridClock initialized: now 1790643336531870 us; error 34 us; skew 500 ppm
Sep 29 00:55:36 dist-test-slave-5309 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:36.540930 31533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:36.541942 31533 webserver.cc:460] Webserver started at http://127.0.0.1:43003/ using document root <none> and password file <none>
I20260929 00:55:36.542513 31533 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:36.542577 31533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:36.542748 31533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:36.544953 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "76a588f668044ea4b1ab86a9962fe308"
format_stamp: "Formatted at 2026-09-29 00:55:36 on dist-test-slave-5309"
server_key: "73d48db0ddd76f183c604e6d8b08cd08"
server_key_iv: "ff8aff689a991e1d3bc386b098676428"
server_key_version: "encryptionkey@0"
I20260929 00:55:36.545533 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "76a588f668044ea4b1ab86a9962fe308"
format_stamp: "Formatted at 2026-09-29 00:55:36 on dist-test-slave-5309"
server_key: "73d48db0ddd76f183c604e6d8b08cd08"
server_key_iv: "ff8aff689a991e1d3bc386b098676428"
server_key_version: "encryptionkey@0"
I20260929 00:55:36.549706 31533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:55:36.552294 31548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:36.553647 31533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:36.553758 31533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "76a588f668044ea4b1ab86a9962fe308"
format_stamp: "Formatted at 2026-09-29 00:55:36 on dist-test-slave-5309"
server_key: "73d48db0ddd76f183c604e6d8b08cd08"
server_key_iv: "ff8aff689a991e1d3bc386b098676428"
server_key_version: "encryptionkey@0"
I20260929 00:55:36.553849 31533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:36.563741 31533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:36.566560 31533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:36.566718 31533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:36.573652 31533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35803
I20260929 00:55:36.573673 31600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35803 every 8 connection(s)
I20260929 00:55:36.574616 31533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:36.577687 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:36.579447 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31533
I20260929 00:55:36.579594 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:36.579910 11427 external_mini_cluster.cc:1428] Setting key 59fea79af7fd4532164a6447a122e722
I20260929 00:55:36.584354 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308: Bootstrap starting.
I20260929 00:55:36.586797 31601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:36.587558 31601 log.cc:824] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:36.589458 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.581514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53493 (local address 127.0.0.1:35803)
0929 00:55:36.582001 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.582017 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.582055 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:36.582733 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.582750 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.583162 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.583524 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.583947 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.585051 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.585978 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.586458 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.586630 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.587495 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.587500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.587513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:36.588009 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.588088 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:36.588123 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.588230 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.588397 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.588627 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260929 00:55:36.589808 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308: No bootstrap required, opened a new log
I20260929 00:55:36.592271 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:36.593713 31601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } }
I20260929 00:55:36.593968 31601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:36.594048 31601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a588f668044ea4b1ab86a9962fe308, State: Initialized, Role: FOLLOWER
I20260929 00:55:36.594543 31601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } }
I20260929 00:55:36.594728 31601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:36.594812 31601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:36.594923 31601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:36.596112 31601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } }
I20260929 00:55:36.596515 31601 leader_election.cc:302] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a588f668044ea4b1ab86a9962fe308; no voters:
I20260929 00:55:36.596882 31601 leader_election.cc:288] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:36.597052 31611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:36.597373 31611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [term 1 LEADER]: Becoming Leader. State: Replica: 76a588f668044ea4b1ab86a9962fe308, State: Running, Role: LEADER
I20260929 00:55:36.597697 31611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } }
I20260929 00:55:36.598300 31601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:36.598915 31613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a588f668044ea4b1ab86a9962fe308. Latest consensus state: current_term: 1 leader_uuid: "76a588f668044ea4b1ab86a9962fe308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } } }
I20260929 00:55:36.599126 31613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:36.599366 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a588f668044ea4b1ab86a9962fe308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a588f668044ea4b1ab86a9962fe308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35803 } } }
I20260929 00:55:36.599464 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:36.599617 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.594400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50871 (local address 127.0.0.1:35803)
0929 00:55:36.594511 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.594514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.594779 (+ 265us) server_negotiation.cc:407] Connection header received
0929 00:55:36.595051 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.595055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.595104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.595187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.595561 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.596133 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.597300 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.597638 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.597715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.598798 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.598802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.598804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:36.598817 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.598845 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:36.598860 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.598939 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.599385 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.599444 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:55:36.599720 31618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:36.603487 31618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:36.609751 31618 catalog_manager.cc:1347] Generated new cluster ID: 12a45fc8436c4633b349dae2223d09c8
I20260929 00:55:36.609865 31618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:36.630599 31618 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:36.631588 31618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:36.638947 31618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a588f668044ea4b1ab86a9962fe308: Generated new TSK 0
I20260929 00:55:36.639693 31618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:36.645423 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31533
2026-09-29T00:55:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:36 dist-test-slave-5309 krb5kdc[31632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:36 dist-test-slave-5309 krb5kdc[31632](info): set up 2 sockets
Sep 29 00:55:36 dist-test-slave-5309 krb5kdc[31632](info): commencing operation
krb5kdc: starting...
W20260929 00:55:37.702603 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:37 dist-test-slave-5309 krb5kdc[31632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:37.842253 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:37.958103 31645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:37.958429 31645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:37.958474 31645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:37.962924 31645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:37.963009 31645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:37.963030 31645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:37.963047 31645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:37.963063 31645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:37.968933 31645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:37.970098 31645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:37.971148 31645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:37.979940 31653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:37.979969 31651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:37.979969 31650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:37.980165 31645 server_base.cc:1034] running on GCE node
I20260929 00:55:37.980911 31645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:37.982050 31645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:37.983240 31645 hybrid_clock.cc:648] HybridClock initialized: now 1790643337983197 us; error 52 us; skew 500 ppm
Sep 29 00:55:37 dist-test-slave-5309 krb5kdc[31632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:37.986253 31645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:37.987136 31645 webserver.cc:460] Webserver started at http://127.0.0.1:37727/ using document root <none> and password file <none>
I20260929 00:55:37.987676 31645 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:37.987746 31645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:37.987927 31645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:37.990061 31645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "f17c1464d80e4c08a2e98a58bb5d4317"
format_stamp: "Formatted at 2026-09-29 00:55:37 on dist-test-slave-5309"
server_key: "438abb5e6443b014193221c1ff525f3c"
server_key_iv: "4df04f13dff283a8157b0738f127a9b9"
server_key_version: "encryptionkey@0"
I20260929 00:55:37.990578 31645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "f17c1464d80e4c08a2e98a58bb5d4317"
format_stamp: "Formatted at 2026-09-29 00:55:37 on dist-test-slave-5309"
server_key: "438abb5e6443b014193221c1ff525f3c"
server_key_iv: "4df04f13dff283a8157b0738f127a9b9"
server_key_version: "encryptionkey@0"
I20260929 00:55:37.994379 31645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:37.996937 31660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:37.998466 31645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:37.998646 31645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "f17c1464d80e4c08a2e98a58bb5d4317"
format_stamp: "Formatted at 2026-09-29 00:55:37 on dist-test-slave-5309"
server_key: "438abb5e6443b014193221c1ff525f3c"
server_key_iv: "4df04f13dff283a8157b0738f127a9b9"
server_key_version: "encryptionkey@0"
I20260929 00:55:37.998786 31645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:38.003777 31645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:38.007277 31645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:38.007489 31645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:38.015239 31645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44883
I20260929 00:55:38.015252 31712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44883 every 8 connection(s)
I20260929 00:55:38.016285 31645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:38.019313 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31645
I20260929 00:55:38.019483 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:38.019548 31713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:38.019800 11427 external_mini_cluster.cc:1428] Setting key 69a091744e699a3e33180bebd5787516
I20260929 00:55:38.026206 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:38.021487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42889 (local address 127.0.0.1:44883)
0929 00:55:38.022022 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:38.022049 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:38.022089 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:55:38.022898 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:38.022917 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:38.023327 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:38.023640 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:38.024875 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:38.024885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:38.024899 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:38.024911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:38.024972 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:38.024993 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:38.025120 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:38.025232 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 00:55:38.025409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260929 00:55:38.027339 31713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317: Bootstrap starting.
I20260929 00:55:38.029105 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:38.029946 31713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:38.030882 31713 log.cc:824] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:38.033314 31713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317: No bootstrap required, opened a new log
I20260929 00:55:38.033524 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:38.031525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38097 (local address 127.0.0.1:44883)
0929 00:55:38.031672 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:38.031677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:38.031928 (+ 251us) server_negotiation.cc:407] Connection header received
0929 00:55:38.032080 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:38.032083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:38.032124 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:38.032212 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:38.033113 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:38.033117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:38.033120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:38.033132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:38.033167 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:38.033178 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:38.033274 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:38.033345 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:55:38.033387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:55:38.037562 31713 raft_consensus.cc:348] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } }
I20260929 00:55:38.037900 31713 raft_consensus.cc:374] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:38.037983 31713 raft_consensus.cc:729] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17c1464d80e4c08a2e98a58bb5d4317, State: Initialized, Role: FOLLOWER
I20260929 00:55:38.038465 31713 consensus_queue.cc:260] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } }
I20260929 00:55:38.038635 31713 raft_consensus.cc:388] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:38.038738 31713 raft_consensus.cc:482] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:38.038856 31713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:38.039963 31713 raft_consensus.cc:504] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } }
I20260929 00:55:38.040452 31713 leader_election.cc:302] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f17c1464d80e4c08a2e98a58bb5d4317; no voters:
I20260929 00:55:38.040886 31713 leader_election.cc:288] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:38.040961 31723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:38.041217 31723 raft_consensus.cc:686] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [term 1 LEADER]: Becoming Leader. State: Replica: f17c1464d80e4c08a2e98a58bb5d4317, State: Running, Role: LEADER
I20260929 00:55:38.041566 31723 consensus_queue.cc:237] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } }
I20260929 00:55:38.042309 31713 sys_catalog.cc:564] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:38.043406 31725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f17c1464d80e4c08a2e98a58bb5d4317. Latest consensus state: current_term: 1 leader_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } } }
I20260929 00:55:38.043558 31725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:38.043898 31724 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17c1464d80e4c08a2e98a58bb5d4317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44883 } } }
I20260929 00:55:38.043962 31732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:38.043994 31724 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:38.047215 31732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:38.053946 31732 catalog_manager.cc:1347] Generated new cluster ID: 58b71addc7ef4221bf1ef7dd58c2569d
I20260929 00:55:38.054041 31732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:38.076100 31732 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:38.076467 31732 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:38.087558 31732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f17c1464d80e4c08a2e98a58bb5d4317: Generated new TSK 0
I20260929 00:55:38.088502 31732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:38.094704 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31645
2026-09-29T00:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:38 dist-test-slave-5309 krb5kdc[31744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:38 dist-test-slave-5309 krb5kdc[31744](info): set up 2 sockets
Sep 29 00:55:38 dist-test-slave-5309 krb5kdc[31744](info): commencing operation
krb5kdc: starting...
W20260929 00:55:39.147020 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:39.285611 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:39.399293 31757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:39.399593 31757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:39.399638 31757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:39.403761 31757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:39.403831 31757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:39.403851 31757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:39.403868 31757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:39.403883 31757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:39.409436 31757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:39.410567 31757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:39.411573 31757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:39.420567 31763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:39.420573 31762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:39.420723 31757 server_base.cc:1034] running on GCE node
W20260929 00:55:39.420573 31765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:39.421603 31757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:39.422891 31757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:39.424118 31757 hybrid_clock.cc:648] HybridClock initialized: now 1790643339424095 us; error 35 us; skew 500 ppm
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:39.427287 31757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:39.428292 31757 webserver.cc:460] Webserver started at http://127.0.0.1:36583/ using document root <none> and password file <none>
I20260929 00:55:39.428973 31757 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:39.429116 31757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:39.429364 31757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:39.431607 31757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "c3bb85092bf94c5388cdd15f2ff0c89e"
format_stamp: "Formatted at 2026-09-29 00:55:39 on dist-test-slave-5309"
server_key: "3865af7c2e2376da997d346137cae6c8"
server_key_iv: "ad1267c0d6ae3efed07800ff86df7f2f"
server_key_version: "encryptionkey@0"
I20260929 00:55:39.432221 31757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "c3bb85092bf94c5388cdd15f2ff0c89e"
format_stamp: "Formatted at 2026-09-29 00:55:39 on dist-test-slave-5309"
server_key: "3865af7c2e2376da997d346137cae6c8"
server_key_iv: "ad1267c0d6ae3efed07800ff86df7f2f"
server_key_version: "encryptionkey@0"
I20260929 00:55:39.437170 31757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 00:55:39.439972 31772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:39.441601 31757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:39.441771 31757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "c3bb85092bf94c5388cdd15f2ff0c89e"
format_stamp: "Formatted at 2026-09-29 00:55:39 on dist-test-slave-5309"
server_key: "3865af7c2e2376da997d346137cae6c8"
server_key_iv: "ad1267c0d6ae3efed07800ff86df7f2f"
server_key_version: "encryptionkey@0"
I20260929 00:55:39.441902 31757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:39.454833 31757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:39.458387 31757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:39.458630 31757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:39.466472 31757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32999
I20260929 00:55:39.466529 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32999 every 8 connection(s)
I20260929 00:55:39.467501 31757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:39.470606 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:39.472973 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31757
I20260929 00:55:39.473130 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:39.473453 11427 external_mini_cluster.cc:1428] Setting key 124f855604095cf0b3571e4b1de0cce2
I20260929 00:55:39.476895 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e: Bootstrap starting.
I20260929 00:55:39.479466 31825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:39.480331 31825 log.cc:824] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:39.482353 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e: No bootstrap required, opened a new log
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:55:39.486738 31825 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260929 00:55:39.487048 31825 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:39.487135 31825 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bb85092bf94c5388cdd15f2ff0c89e, State: Initialized, Role: FOLLOWER
I20260929 00:55:39.487649 31825 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260929 00:55:39.487820 31825 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:39.487885 31825 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:39.488009 31825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:39.489276 31825 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260929 00:55:39.489636 31825 leader_election.cc:302] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3bb85092bf94c5388cdd15f2ff0c89e; no voters:
I20260929 00:55:39.489898 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:39.474970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:47549 (local address 127.0.0.1:32999)
0929 00:55:39.475411 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:39.475436 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:39.475471 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:55:39.476128 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:39.476145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:39.476492 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:39.476783 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:39.477780 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:39.478878 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:39.480069 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:39.480594 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:39.480774 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:39.483390 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:39.483396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:39.483413 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:39.483444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:39.486196 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:39.486743 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:39.486751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:39.486760 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:39.486895 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:39.487415 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:39.487419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:39.487421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:39.487956 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:39.488193 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:39.488581 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 00:55:39.488830 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":157,"threads_started":1}
I20260929 00:55:39.490043 31825 leader_election.cc:288] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:39.490283 31830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:39.490521 31830 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [term 1 LEADER]: Becoming Leader. State: Replica: c3bb85092bf94c5388cdd15f2ff0c89e, State: Running, Role: LEADER
I20260929 00:55:39.490875 31830 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260929 00:55:39.491376 31825 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:39.492630 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } } }
I20260929 00:55:39.492780 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:39.493122 31832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3bb85092bf94c5388cdd15f2ff0c89e. Latest consensus state: current_term: 1 leader_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3bb85092bf94c5388cdd15f2ff0c89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } } }
I20260929 00:55:39.493218 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:39.493223 31832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:39.493220 31839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:39.496690 31839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:39.504243 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:39.495684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:55449 (local address 127.0.0.1:32999)
0929 00:55:39.495792 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:39.495796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:39.495958 (+ 162us) server_negotiation.cc:407] Connection header received
0929 00:55:39.496086 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:39.496089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:39.496140 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:39.496228 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:39.497335 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:39.497983 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:39.498848 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:39.499114 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:39.499228 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:39.500746 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:39.500749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:39.500752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:39.500780 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:39.502367 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:39.502864 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:39.502867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:39.502869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:39.502960 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:39.503331 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:39.503334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:39.503336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:39.503537 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:39.503698 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:39.503976 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 00:55:39.504073 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 00:55:39.506706 31839 catalog_manager.cc:1347] Generated new cluster ID: 7aa5556816f944d4bbefd879052f9476
I20260929 00:55:39.506798 31839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:39.527238 31839 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:39.528571 31839 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:39.541275 31839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3bb85092bf94c5388cdd15f2ff0c89e: Generated new TSK 0
I20260929 00:55:39.542243 31839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:39.548144 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31757
W20260929 00:55:39.553927 31742 connection.cc:535] client connection to 127.0.0.1:32999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32999): Connection reset by peer (error 104)
2026-09-29T00:55:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31856](info): set up 2 sockets
Sep 29 00:55:39 dist-test-slave-5309 krb5kdc[31856](info): commencing operation
krb5kdc: starting...
W20260929 00:55:40.606802 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:40 dist-test-slave-5309 krb5kdc[31856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:40.738451 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:40.855031 31869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:40.855434 31869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:40.855556 31869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:40.859706 31869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:40.859804 31869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:40.859834 31869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:40.859863 31869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:40.859898 31869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:40.865368 31869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:40.866518 31869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:40.867581 31869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:40.876482 31875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:40.876529 31877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:40.876504 31869 server_base.cc:1034] running on GCE node
W20260929 00:55:40.876703 31874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:40.877434 31869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:40.878643 31869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:40.879849 31869 hybrid_clock.cc:648] HybridClock initialized: now 1790643340879832 us; error 31 us; skew 500 ppm
Sep 29 00:55:40 dist-test-slave-5309 krb5kdc[31856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:40.882659 31869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:40.883580 31869 webserver.cc:460] Webserver started at http://127.0.0.1:41699/ using document root <none> and password file <none>
I20260929 00:55:40.884166 31869 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:40.884260 31869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:40.884500 31869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:40.886690 31869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "abcd58875d7e4d39a0360746307aaced"
format_stamp: "Formatted at 2026-09-29 00:55:40 on dist-test-slave-5309"
server_key: "bb1b7365e5acd52a2fa030ed2889f79b"
server_key_iv: "d7613eadf11341d52a49f10ab47e6c69"
server_key_version: "encryptionkey@0"
I20260929 00:55:40.887326 31869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "abcd58875d7e4d39a0360746307aaced"
format_stamp: "Formatted at 2026-09-29 00:55:40 on dist-test-slave-5309"
server_key: "bb1b7365e5acd52a2fa030ed2889f79b"
server_key_iv: "d7613eadf11341d52a49f10ab47e6c69"
server_key_version: "encryptionkey@0"
I20260929 00:55:40.891389 31869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:40.894162 31884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:40.895562 31869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:40.895730 31869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "abcd58875d7e4d39a0360746307aaced"
format_stamp: "Formatted at 2026-09-29 00:55:40 on dist-test-slave-5309"
server_key: "bb1b7365e5acd52a2fa030ed2889f79b"
server_key_iv: "d7613eadf11341d52a49f10ab47e6c69"
server_key_version: "encryptionkey@0"
I20260929 00:55:40.895864 31869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:40.911749 31869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:40.915148 31869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:40.915392 31869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:40.923251 31869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35327
I20260929 00:55:40.923277 31936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35327 every 8 connection(s)
I20260929 00:55:40.924230 31869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:40.925871 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31869
I20260929 00:55:40.926024 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:40.926376 11427 external_mini_cluster.cc:1428] Setting key 9131594fcf86ff00058a1ac702a3ddb1
I20260929 00:55:40.927824 31937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:40.935391 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced: Bootstrap starting.
I20260929 00:55:40.937155 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:40.927822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:51731 (local address 127.0.0.1:35327)
0929 00:55:40.928304 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:40.928324 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:40.928371 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:55:40.929298 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:40.929322 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:40.929787 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:40.930176 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:40.930619 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:40.932188 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:40.933195 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:40.933921 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:40.934115 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:40.935013 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:40.935019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:40.935037 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:40.935562 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:40.935645 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:40.935678 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:40.935823 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:40.936013 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 00:55:40.936152 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":116,"threads_started":1}
I20260929 00:55:40.938172 31937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:40.939077 31937 log.cc:824] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:40.940515 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:40.941614 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced: No bootstrap required, opened a new log
I20260929 00:55:40.945482 31937 raft_consensus.cc:348] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } }
I20260929 00:55:40.945756 31937 raft_consensus.cc:374] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:40.945819 31937 raft_consensus.cc:729] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcd58875d7e4d39a0360746307aaced, State: Initialized, Role: FOLLOWER
I20260929 00:55:40.946316 31937 consensus_queue.cc:260] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } }
I20260929 00:55:40.946472 31937 raft_consensus.cc:388] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:40.946560 31937 raft_consensus.cc:482] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:40.946666 31937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:40.946925 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:40.942724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:35593 (local address 127.0.0.1:35327)
0929 00:55:40.942848 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:40.942852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:40.943023 (+ 171us) server_negotiation.cc:407] Connection header received
0929 00:55:40.943179 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:40.943182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:40.943232 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:40.943319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:40.943627 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:40.944153 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:40.945079 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:40.945326 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:40.945448 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:40.946424 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:40.946428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:40.946431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:40.946443 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:40.946473 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:40.946485 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:40.946569 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:40.946729 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:55:40.946776 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:55:40.947929 31937 raft_consensus.cc:504] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } }
I20260929 00:55:40.948325 31937 leader_election.cc:302] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abcd58875d7e4d39a0360746307aaced; no voters:
I20260929 00:55:40.948588 31937 leader_election.cc:288] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:40.948731 31947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:40.948971 31947 raft_consensus.cc:686] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [term 1 LEADER]: Becoming Leader. State: Replica: abcd58875d7e4d39a0360746307aaced, State: Running, Role: LEADER
I20260929 00:55:40.949317 31947 consensus_queue.cc:237] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } }
I20260929 00:55:40.949878 31937 sys_catalog.cc:564] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:40.950973 31949 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [sys.catalog]: SysCatalogTable state changed. Reason: New leader abcd58875d7e4d39a0360746307aaced. Latest consensus state: current_term: 1 leader_uuid: "abcd58875d7e4d39a0360746307aaced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } } }
I20260929 00:55:40.951116 31949 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:40.951526 31956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:40.951503 31948 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abcd58875d7e4d39a0360746307aaced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcd58875d7e4d39a0360746307aaced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35327 } } }
I20260929 00:55:40.951671 31948 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:40.954797 31956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:40.961689 31956 catalog_manager.cc:1347] Generated new cluster ID: 2df3ff4b33bb45d89ecec452c8edf522
I20260929 00:55:40.961797 31956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:40.979334 31956 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:40.980669 31956 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:40.987811 31956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abcd58875d7e4d39a0360746307aaced: Generated new TSK 0
I20260929 00:55:40.988587 31956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:40.993840 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31869
W20260929 00:55:40.998875 31854 connection.cc:535] client connection to 127.0.0.1:35327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35327): Connection reset by peer (error 104)
2026-09-29T00:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:41 dist-test-slave-5309 krb5kdc[31968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:41 dist-test-slave-5309 krb5kdc[31968](info): set up 2 sockets
Sep 29 00:55:41 dist-test-slave-5309 krb5kdc[31968](info): commencing operation
krb5kdc: starting...
W20260929 00:55:42.090524 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:42 dist-test-slave-5309 krb5kdc[31968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:42.226259 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:42.339166 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:42.339515 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:42.339602 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:42.343616 31981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:42.343716 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:42.343755 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:42.343786 31981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:42.343816 31981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:42.349212 31981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:42.350274 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:42.351330 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:42.360265 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:42.360298 31981 server_base.cc:1034] running on GCE node
W20260929 00:55:42.360369 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:42.360260 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:42.361121 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:42.362329 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:42.363528 31981 hybrid_clock.cc:648] HybridClock initialized: now 1790643342363504 us; error 40 us; skew 500 ppm
Sep 29 00:55:42 dist-test-slave-5309 krb5kdc[31968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:42.370986 31981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:42.371999 31981 webserver.cc:460] Webserver started at http://127.0.0.1:38285/ using document root <none> and password file <none>
I20260929 00:55:42.372592 31981 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:42.372686 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:42.372958 31981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:42.375087 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "e285fae4bcbb4a8c8e0affb988f23638"
format_stamp: "Formatted at 2026-09-29 00:55:42 on dist-test-slave-5309"
server_key: "541940d098f707ae97bcf7a436e9761d"
server_key_iv: "9fa1567cae651f83bc83687d87d965d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:42.375716 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "e285fae4bcbb4a8c8e0affb988f23638"
format_stamp: "Formatted at 2026-09-29 00:55:42 on dist-test-slave-5309"
server_key: "541940d098f707ae97bcf7a436e9761d"
server_key_iv: "9fa1567cae651f83bc83687d87d965d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:42.379909 31981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:42.382617 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:42.383929 31981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:55:42.384079 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "e285fae4bcbb4a8c8e0affb988f23638"
format_stamp: "Formatted at 2026-09-29 00:55:42 on dist-test-slave-5309"
server_key: "541940d098f707ae97bcf7a436e9761d"
server_key_iv: "9fa1567cae651f83bc83687d87d965d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:42.384202 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:42.389005 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:42.392012 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:42.392223 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:42.399610 31981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38411
I20260929 00:55:42.399623 32048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38411 every 8 connection(s)
I20260929 00:55:42.400601 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:42.403344 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 31981
I20260929 00:55:42.403455 32049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:42.403512 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:42.403820 11427 external_mini_cluster.cc:1428] Setting key 7e336afab2dd2d84bd96dd8e1cc35c37
I20260929 00:55:42.410115 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:42.405365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:53347 (local address 127.0.0.1:38411)
0929 00:55:42.405836 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:42.405856 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:42.405900 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:55:42.407115 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:42.407142 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:42.407527 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:42.407946 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:42.408669 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:42.408682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:42.408698 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:42.408709 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:42.408772 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:42.408797 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:42.408964 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:42.409033 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 00:55:42.409193 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":131,"threads_started":1}
I20260929 00:55:42.411193 32049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638: Bootstrap starting.
I20260929 00:55:42.412985 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:42.413695 32049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:42.414489 32049 log.cc:824] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:42.416908 32049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638: No bootstrap required, opened a new log
I20260929 00:55:42.417199 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:42.415236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:50289 (local address 127.0.0.1:38411)
0929 00:55:42.415351 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:42.415358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:42.415621 (+ 263us) server_negotiation.cc:407] Connection header received
0929 00:55:42.415746 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:42.415748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:42.415777 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:42.415843 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:42.416654 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:42.416657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:42.416659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:42.416671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:42.416706 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:42.416716 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:42.416795 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:42.417054 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 00:55:42.417110 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:55:42.421005 32049 raft_consensus.cc:348] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260929 00:55:42.421301 32049 raft_consensus.cc:374] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:42.421434 32049 raft_consensus.cc:729] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e285fae4bcbb4a8c8e0affb988f23638, State: Initialized, Role: FOLLOWER
I20260929 00:55:42.421875 32049 consensus_queue.cc:260] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260929 00:55:42.422025 32049 raft_consensus.cc:388] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:42.422094 32049 raft_consensus.cc:482] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:42.422191 32049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:42.423187 32049 raft_consensus.cc:504] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260929 00:55:42.423554 32049 leader_election.cc:302] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e285fae4bcbb4a8c8e0affb988f23638; no voters:
I20260929 00:55:42.423885 32049 leader_election.cc:288] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:42.424068 32059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:42.424324 32059 raft_consensus.cc:686] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [term 1 LEADER]: Becoming Leader. State: Replica: e285fae4bcbb4a8c8e0affb988f23638, State: Running, Role: LEADER
I20260929 00:55:42.424683 32059 consensus_queue.cc:237] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260929 00:55:42.425257 32049 sys_catalog.cc:564] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:42.426501 32060 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e285fae4bcbb4a8c8e0affb988f23638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260929 00:55:42.426647 32060 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:42.426739 32061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e285fae4bcbb4a8c8e0affb988f23638. Latest consensus state: current_term: 1 leader_uuid: "e285fae4bcbb4a8c8e0affb988f23638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285fae4bcbb4a8c8e0affb988f23638" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260929 00:55:42.426846 32061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:42.427150 32068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:42.430518 32068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:42.436566 32068 catalog_manager.cc:1347] Generated new cluster ID: 3cb0f78a65ae428bae93ae9769914080
I20260929 00:55:42.436646 32068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:42.462070 32068 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:42.462525 32068 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:42.473062 32068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e285fae4bcbb4a8c8e0affb988f23638: Generated new TSK 0
I20260929 00:55:42.473806 32068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:42.488729 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 31981
2026-09-29T00:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:42 dist-test-slave-5309 krb5kdc[32080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:42 dist-test-slave-5309 krb5kdc[32080](info): set up 2 sockets
Sep 29 00:55:42 dist-test-slave-5309 krb5kdc[32080](info): commencing operation
krb5kdc: starting...
W20260929 00:55:43.542428 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:43 dist-test-slave-5309 krb5kdc[32080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:43.678174 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:43.797103 32093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:43.797582 32093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:43.797679 32093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:43.802459 32093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:43.802534 32093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:43.802554 32093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:43.802572 32093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:43.802587 32093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:43.808424 32093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:43.809655 32093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:43.810743 32093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:43.820206 32098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:43.820310 32101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:43.820328 32099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:43.820389 32093 server_base.cc:1034] running on GCE node
I20260929 00:55:43.821115 32093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:43.822286 32093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:43.823494 32093 hybrid_clock.cc:648] HybridClock initialized: now 1790643343823458 us; error 46 us; skew 500 ppm
Sep 29 00:55:43 dist-test-slave-5309 krb5kdc[32080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:43.826383 32093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:43.827329 32093 webserver.cc:460] Webserver started at http://127.0.0.1:44283/ using document root <none> and password file <none>
I20260929 00:55:43.827965 32093 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:43.828060 32093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:43.828302 32093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:43.830562 32093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "fd9f174f2889479cb580ecc9fdffc4ef"
format_stamp: "Formatted at 2026-09-29 00:55:43 on dist-test-slave-5309"
server_key: "a4ff5bfd10a55eb60f063735fdbc7350"
server_key_iv: "57cbf63bc3ddf5a95df8f8dfb4c61471"
server_key_version: "encryptionkey@0"
I20260929 00:55:43.831192 32093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "fd9f174f2889479cb580ecc9fdffc4ef"
format_stamp: "Formatted at 2026-09-29 00:55:43 on dist-test-slave-5309"
server_key: "a4ff5bfd10a55eb60f063735fdbc7350"
server_key_iv: "57cbf63bc3ddf5a95df8f8dfb4c61471"
server_key_version: "encryptionkey@0"
I20260929 00:55:43.835326 32093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:43.838032 32108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:43.839586 32093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:43.839743 32093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "fd9f174f2889479cb580ecc9fdffc4ef"
format_stamp: "Formatted at 2026-09-29 00:55:43 on dist-test-slave-5309"
server_key: "a4ff5bfd10a55eb60f063735fdbc7350"
server_key_iv: "57cbf63bc3ddf5a95df8f8dfb4c61471"
server_key_version: "encryptionkey@0"
I20260929 00:55:43.839877 32093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:43.866696 32093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:43.870221 32093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:43.870465 32093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:43.877954 32093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44315
I20260929 00:55:43.877980 32160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44315 every 8 connection(s)
I20260929 00:55:43.878963 32093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:43.881892 32161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:43.885627 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32093
I20260929 00:55:43.885784 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:43.886073 11427 external_mini_cluster.cc:1428] Setting key 8ed571d73a8f749c252c1d1fd796597a
I20260929 00:55:43.888487 32161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef: Bootstrap starting.
I20260929 00:55:43.890925 32161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:43.891733 32161 log.cc:824] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:43.893980 32161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef: No bootstrap required, opened a new log
Sep 29 00:55:43 dist-test-slave-5309 krb5kdc[32080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:55:43.898522 32161 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260929 00:55:43.898818 32161 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:43.898907 32161 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9f174f2889479cb580ecc9fdffc4ef, State: Initialized, Role: FOLLOWER
I20260929 00:55:43.899447 32161 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260929 00:55:43.899606 32161 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:43.899710 32161 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:43.899816 32161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:43.900853 32161 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260929 00:55:43.901290 32161 leader_election.cc:302] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9f174f2889479cb580ecc9fdffc4ef; no voters:
I20260929 00:55:43.901783 32166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:43.901706 32161 leader_election.cc:288] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:43.902015 32166 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [term 1 LEADER]: Becoming Leader. State: Replica: fd9f174f2889479cb580ecc9fdffc4ef, State: Running, Role: LEADER
I20260929 00:55:43.902410 32166 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260929 00:55:43.902532 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:43.887635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:58925 (local address 127.0.0.1:44315)
0929 00:55:43.888136 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:43.888151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:43.888234 (+ 83us) server_negotiation.cc:407] Connection header received
0929 00:55:43.889030 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:43.889051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:43.889423 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:43.889775 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:43.890598 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:43.891690 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:43.892876 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:43.893361 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:43.893539 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:43.896639 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:43.896645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:43.896661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:43.896693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:43.898857 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:43.899489 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:43.899495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:43.899500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:43.899659 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:43.900047 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:43.900050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:43.900052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:43.900475 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:43.900730 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:43.901072 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 00:55:43.901318 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":166,"threads_started":1}
I20260929 00:55:43.903177 32161 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:43.903644 32166 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260929 00:55:43.903836 32166 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:43.904055 32167 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9f174f2889479cb580ecc9fdffc4ef. Latest consensus state: current_term: 1 leader_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9f174f2889479cb580ecc9fdffc4ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260929 00:55:43.904152 32167 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:43.905105 32170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:43.907009 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:43.908759 32170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:43.918035 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:43.909706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:38645 (local address 127.0.0.1:44315)
0929 00:55:43.909821 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:43.909827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:43.909842 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:43.909880 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:43.909883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:43.909934 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:43.910023 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:43.910862 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:43.911604 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:43.912429 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:43.912706 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:43.912864 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:43.914522 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:43.914526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:43.914529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:43.914555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:43.916072 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:43.916600 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:43.916604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:43.916605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:43.916692 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:43.917101 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:43.917104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:43.917106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:43.917295 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:43.917449 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:43.917727 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 00:55:43.917831 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 00:55:43.918732 32170 catalog_manager.cc:1347] Generated new cluster ID: 14350733245840439a24b6661ce12344
I20260929 00:55:43.918848 32170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:43.953500 32170 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:43.954842 32170 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:43.965627 32170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9f174f2889479cb580ecc9fdffc4ef: Generated new TSK 0
I20260929 00:55:43.966609 32170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:43.983233 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32093
W20260929 00:55:43.989125 32078 connection.cc:535] client connection to 127.0.0.1:44315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44315): Connection reset by peer (error 104)
2026-09-29T00:55:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:44 dist-test-slave-5309 krb5kdc[32192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:44 dist-test-slave-5309 krb5kdc[32192](info): set up 2 sockets
Sep 29 00:55:44 dist-test-slave-5309 krb5kdc[32192](info): commencing operation
krb5kdc: starting...
W20260929 00:55:45.042658 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:45 dist-test-slave-5309 krb5kdc[32192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:45.174240 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:45.289011 32205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:45.289338 32205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:45.289428 32205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:45.293627 32205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:45.293728 32205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:45.293764 32205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:45.293823 32205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:45.293879 32205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:45.299419 32205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:45.300595 32205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:45.301759 32205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:45.310812 32210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:45.310828 32213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:45.310922 32211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:45.310912 32205 server_base.cc:1034] running on GCE node
I20260929 00:55:45.311571 32205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:45.312709 32205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:45.313910 32205 hybrid_clock.cc:648] HybridClock initialized: now 1790643345313882 us; error 40 us; skew 500 ppm
Sep 29 00:55:45 dist-test-slave-5309 krb5kdc[32192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:45.316686 32205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:45.317587 32205 webserver.cc:460] Webserver started at http://127.0.0.1:38613/ using document root <none> and password file <none>
I20260929 00:55:45.318114 32205 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:45.318176 32205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:45.318346 32205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:45.320413 32205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "82348e76508b44bf867e42170b3c220b"
format_stamp: "Formatted at 2026-09-29 00:55:45 on dist-test-slave-5309"
server_key: "302ea0b717d6d0c6dd2ebcc7f444fae4"
server_key_iv: "de8146bc4ec129c66e2f8fddd64dd014"
server_key_version: "encryptionkey@0"
I20260929 00:55:45.321024 32205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "82348e76508b44bf867e42170b3c220b"
format_stamp: "Formatted at 2026-09-29 00:55:45 on dist-test-slave-5309"
server_key: "302ea0b717d6d0c6dd2ebcc7f444fae4"
server_key_iv: "de8146bc4ec129c66e2f8fddd64dd014"
server_key_version: "encryptionkey@0"
I20260929 00:55:45.325026 32205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:55:45.327503 32220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:45.328950 32205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:45.329087 32205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "82348e76508b44bf867e42170b3c220b"
format_stamp: "Formatted at 2026-09-29 00:55:45 on dist-test-slave-5309"
server_key: "302ea0b717d6d0c6dd2ebcc7f444fae4"
server_key_iv: "de8146bc4ec129c66e2f8fddd64dd014"
server_key_version: "encryptionkey@0"
I20260929 00:55:45.329205 32205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:45.356699 32205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:45.359965 32205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:45.360205 32205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:45.367563 32205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32883
I20260929 00:55:45.367604 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32883 every 8 connection(s)
I20260929 00:55:45.368603 32205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:45.371450 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32205
I20260929 00:55:45.371485 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:45.371559 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:45.371898 11427 external_mini_cluster.cc:1428] Setting key 1a048a9d3dfcfaecf70496edde6ed0ce
I20260929 00:55:45.378278 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b: Bootstrap starting.
I20260929 00:55:45.380852 32273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:45.381232 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:45.373407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:54185 (local address 127.0.0.1:32883)
0929 00:55:45.373853 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:45.373868 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:45.373904 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:55:45.374586 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:45.374603 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:45.374952 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:45.375236 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:45.375569 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:45.376575 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:45.377807 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:45.378294 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:45.378473 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:45.379350 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:45.379355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:45.379367 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:45.379775 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:45.379828 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:45.379851 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:45.379961 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:45.380161 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:55:45.380340 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":153,"threads_started":1}
I20260929 00:55:45.381657 32273 log.cc:824] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:45.383713 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b: No bootstrap required, opened a new log
I20260929 00:55:45.384447 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:45.388000 32273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 00:55:45.388291 32273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:45.388366 32273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82348e76508b44bf867e42170b3c220b, State: Initialized, Role: FOLLOWER
I20260929 00:55:45.388844 32273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 00:55:45.389016 32273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:45.389092 32273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:45.389196 32273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:45.390506 32273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 00:55:45.390832 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:45.386623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:53331 (local address 127.0.0.1:32883)
0929 00:55:45.386741 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:45.386745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:45.386873 (+ 128us) server_negotiation.cc:407] Connection header received
0929 00:55:45.386987 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:45.386991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:45.387030 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:45.387097 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:45.387385 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:45.387902 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:45.389007 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:45.389390 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:45.389473 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:45.390310 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:45.390313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:45.390315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:45.390326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:45.390349 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:45.390359 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:45.390430 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:45.390642 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 00:55:45.390681 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:55:45.390879 32273 leader_election.cc:302] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82348e76508b44bf867e42170b3c220b; no voters:
I20260929 00:55:45.391243 32273 leader_election.cc:288] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:45.391383 32283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:45.391582 32283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [term 1 LEADER]: Becoming Leader. State: Replica: 82348e76508b44bf867e42170b3c220b, State: Running, Role: LEADER
I20260929 00:55:45.391891 32283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 00:55:45.392604 32273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:45.393652 32285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82348e76508b44bf867e42170b3c220b. Latest consensus state: current_term: 1 leader_uuid: "82348e76508b44bf867e42170b3c220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } } }
I20260929 00:55:45.393774 32285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:45.394150 32292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:45.394186 32284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82348e76508b44bf867e42170b3c220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82348e76508b44bf867e42170b3c220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } } }
I20260929 00:55:45.394363 32284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:45.396953 32292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:45.402752 32292 catalog_manager.cc:1347] Generated new cluster ID: f5310c13a8da480fbb78df7766122067
I20260929 00:55:45.402835 32292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:45.416525 32292 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:45.417570 32292 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:45.431800 32292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82348e76508b44bf867e42170b3c220b: Generated new TSK 0
I20260929 00:55:45.432631 32292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:45.436756 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32205
W20260929 00:55:45.441712 32190 connection.cc:535] client connection to 127.0.0.1:32883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32883): Connection reset by peer (error 104)
2026-09-29T00:55:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:45 dist-test-slave-5309 krb5kdc[32304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:45 dist-test-slave-5309 krb5kdc[32304](info): set up 2 sockets
Sep 29 00:55:45 dist-test-slave-5309 krb5kdc[32304](info): commencing operation
krb5kdc: starting...
W20260929 00:55:46.490634 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:46 dist-test-slave-5309 krb5kdc[32304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:55:46.616060 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:46.736685 32317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:46.737097 32317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:46.737196 32317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:46.741302 32317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:46.741391 32317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:46.741423 32317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:46.741464 32317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:46.741500 32317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:46.746917 32317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:46.748081 32317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:46.749197 32317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:46.758329 32325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:46.758495 32317 server_base.cc:1034] running on GCE node
W20260929 00:55:46.758390 32323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:46.758316 32322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:46.759357 32317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:46.760536 32317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:46.761765 32317 hybrid_clock.cc:648] HybridClock initialized: now 1790643346761741 us; error 35 us; skew 500 ppm
Sep 29 00:55:46 dist-test-slave-5309 krb5kdc[32304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:46.764943 32317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:55:46.766044 32317 webserver.cc:460] Webserver started at http://127.0.0.1:38397/ using document root <none> and password file <none>
I20260929 00:55:46.766677 32317 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:46.766783 32317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:46.767117 32317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:46.769498 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "695adbf6349142eeb41ddd02426a5f3d"
format_stamp: "Formatted at 2026-09-29 00:55:46 on dist-test-slave-5309"
server_key: "63ad1302b12144ae6bb50da52176251e"
server_key_iv: "7c3387af75517594fab4b1da6ac2cba3"
server_key_version: "encryptionkey@0"
I20260929 00:55:46.770117 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "695adbf6349142eeb41ddd02426a5f3d"
format_stamp: "Formatted at 2026-09-29 00:55:46 on dist-test-slave-5309"
server_key: "63ad1302b12144ae6bb50da52176251e"
server_key_iv: "7c3387af75517594fab4b1da6ac2cba3"
server_key_version: "encryptionkey@0"
I20260929 00:55:46.774127 32317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:46.776703 32332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:46.778136 32317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:46.778347 32317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "695adbf6349142eeb41ddd02426a5f3d"
format_stamp: "Formatted at 2026-09-29 00:55:46 on dist-test-slave-5309"
server_key: "63ad1302b12144ae6bb50da52176251e"
server_key_iv: "7c3387af75517594fab4b1da6ac2cba3"
server_key_version: "encryptionkey@0"
I20260929 00:55:46.778478 32317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:46.783344 32317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:46.786787 32317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:46.787007 32317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:46.794715 32317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41355
I20260929 00:55:46.794780 32384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41355 every 8 connection(s)
I20260929 00:55:46.795748 32317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:46.798751 32385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:46.803241 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32317
I20260929 00:55:46.803426 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:46.803714 11427 external_mini_cluster.cc:1428] Setting key 498739289b0b6e84419f278f0b5c0f34
I20260929 00:55:46.804695 32385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d: Bootstrap starting.
I20260929 00:55:46.807512 32385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:46.808605 32385 log.cc:824] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:46.809650 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:46.805190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:60445 (local address 127.0.0.1:41355)
0929 00:55:46.805674 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:46.805701 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:46.805744 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:55:46.806612 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:46.806636 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:46.807007 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:46.807366 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:46.808168 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:46.808174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:46.808190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:46.808201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:46.808261 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:46.808286 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:46.808400 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:46.808560 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:55:46.808739 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260929 00:55:46.810752 32385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d: No bootstrap required, opened a new log
I20260929 00:55:46.812542 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:46.815327 32385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 00:55:46.815667 32385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:46.815742 32385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695adbf6349142eeb41ddd02426a5f3d, State: Initialized, Role: FOLLOWER
I20260929 00:55:46.816285 32385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 00:55:46.816478 32385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:46.816556 32385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:46.816665 32385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:46.817126 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:46.815094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:45137 (local address 127.0.0.1:41355)
0929 00:55:46.815236 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:46.815240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:46.815418 (+ 178us) server_negotiation.cc:407] Connection header received
0929 00:55:46.815510 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:46.815513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:46.815550 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:46.815651 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:46.816654 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:46.816664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:46.816666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:46.816677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:46.816704 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:46.816715 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:46.816854 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:46.816952 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 00:55:46.817000 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:55:46.818266 32385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 00:55:46.818650 32385 leader_election.cc:302] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695adbf6349142eeb41ddd02426a5f3d; no voters:
I20260929 00:55:46.819020 32385 leader_election.cc:288] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:46.819170 32395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:46.819439 32395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [term 1 LEADER]: Becoming Leader. State: Replica: 695adbf6349142eeb41ddd02426a5f3d, State: Running, Role: LEADER
I20260929 00:55:46.819834 32395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 00:55:46.820343 32385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:46.821658 32396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695adbf6349142eeb41ddd02426a5f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260929 00:55:46.821792 32396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:46.822165 32397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695adbf6349142eeb41ddd02426a5f3d. Latest consensus state: current_term: 1 leader_uuid: "695adbf6349142eeb41ddd02426a5f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695adbf6349142eeb41ddd02426a5f3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260929 00:55:46.822229 32404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:46.822296 32397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:46.825572 32404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:46.832080 32404 catalog_manager.cc:1347] Generated new cluster ID: 8eccad8655fb455283b2e2c000e42d3c
I20260929 00:55:46.832175 32404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:46.846112 32404 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:46.846468 32404 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:46.854033 32404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695adbf6349142eeb41ddd02426a5f3d: Generated new TSK 0
I20260929 00:55:46.854781 32404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:46.865852 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32317
2026-09-29T00:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:46 dist-test-slave-5309 krb5kdc[32416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:46 dist-test-slave-5309 krb5kdc[32416](info): set up 2 sockets
Sep 29 00:55:46 dist-test-slave-5309 krb5kdc[32416](info): commencing operation
krb5kdc: starting...
W20260929 00:55:47.918560 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:48.046104 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33897
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:48.161751 32429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:48.162108 32429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:48.162154 32429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:48.166124 32429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:48.166185 32429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:48.166204 32429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:48.166220 32429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:48.166235 32429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:48.171506 32429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:48.172610 32429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:48.173621 32429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:48.182547 32437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:48.182542 32435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:48.182547 32434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:48.182587 32429 server_base.cc:1034] running on GCE node
I20260929 00:55:48.183465 32429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:48.184626 32429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:48.185842 32429 hybrid_clock.cc:648] HybridClock initialized: now 1790643348185831 us; error 34 us; skew 500 ppm
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:48.194900 32429 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:48.195952 32429 webserver.cc:460] Webserver started at http://127.11.40.254:43625/ using document root <none> and password file <none>
I20260929 00:55:48.196545 32429 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:48.196640 32429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:48.196954 32429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:48.199080 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "12e7bd613b90496b8ee40db11a5a6c0e"
format_stamp: "Formatted at 2026-09-29 00:55:48 on dist-test-slave-5309"
server_key: "07af9cb0bcfd6f84e8ffed3ef7463d06"
server_key_iv: "80e4eb8380d8309b01cee139d73bd5d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:48.199679 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "12e7bd613b90496b8ee40db11a5a6c0e"
format_stamp: "Formatted at 2026-09-29 00:55:48 on dist-test-slave-5309"
server_key: "07af9cb0bcfd6f84e8ffed3ef7463d06"
server_key_iv: "80e4eb8380d8309b01cee139d73bd5d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:48.203514 32429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:48.206066 32444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:48.207831 32429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:48.207993 32429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "12e7bd613b90496b8ee40db11a5a6c0e"
format_stamp: "Formatted at 2026-09-29 00:55:48 on dist-test-slave-5309"
server_key: "07af9cb0bcfd6f84e8ffed3ef7463d06"
server_key_iv: "80e4eb8380d8309b01cee139d73bd5d0"
server_key_version: "encryptionkey@0"
I20260929 00:55:48.208101 32429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:48.230921 32429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:48.234266 32429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:48.234433 32429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:48.241967 32429 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33897
I20260929 00:55:48.241994 32496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33897 every 8 connection(s)
I20260929 00:55:48.242997 32429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:48.243381 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32429
I20260929 00:55:48.243546 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:48.243829 11427 external_mini_cluster.cc:1428] Setting key 2d85b69a96d745aec2d5c714dd6c172c
I20260929 00:55:48.246797 32497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:48.254045 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e: Bootstrap starting.
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:48.256947 32497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:48.257877 32497 log.cc:824] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:48.260315 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e: No bootstrap required, opened a new log
I20260929 00:55:48.260775 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:48.245532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58733 (local address 127.11.40.254:33897)
0929 00:55:48.245982 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:48.246001 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:48.246043 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:55:48.246916 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:48.246940 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:48.247393 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:48.247768 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:48.248507 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:48.250035 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:48.250938 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:48.251613 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:48.251790 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:48.255102 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:48.255107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:48.255122 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:48.255155 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:48.257245 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:48.257895 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:48.257900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:48.257905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:48.258084 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:48.258483 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:48.258487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:48.258489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:48.259039 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:48.259225 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:48.259578 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 00:55:48.259872 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":118,"threads_started":1}
I20260929 00:55:48.264133 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:48.264293 32497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } }
I20260929 00:55:48.264539 32497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:48.264618 32497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e7bd613b90496b8ee40db11a5a6c0e, State: Initialized, Role: FOLLOWER
I20260929 00:55:48.265137 32497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } }
I20260929 00:55:48.265269 32497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:48.265358 32497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:48.265461 32497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:48.266536 32497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } }
I20260929 00:55:48.266884 32497 leader_election.cc:302] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e7bd613b90496b8ee40db11a5a6c0e; no voters:
I20260929 00:55:48.267221 32497 leader_election.cc:288] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:48.267293 32507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:48.267530 32507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [term 1 LEADER]: Becoming Leader. State: Replica: 12e7bd613b90496b8ee40db11a5a6c0e, State: Running, Role: LEADER
I20260929 00:55:48.267990 32507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } }
I20260929 00:55:48.268710 32497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:48.269189 32508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } } }
I20260929 00:55:48.269304 32508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:48.269960 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e7bd613b90496b8ee40db11a5a6c0e. Latest consensus state: current_term: 1 leader_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e7bd613b90496b8ee40db11a5a6c0e" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33897 } } }
I20260929 00:55:48.270054 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:48.270912 32511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:48.274037 32511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:48.277266 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:48.266653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39875 (local address 127.11.40.254:33897)
0929 00:55:48.266787 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:48.266792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:48.267032 (+ 240us) server_negotiation.cc:407] Connection header received
0929 00:55:48.267215 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:48.267218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:48.267304 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:48.267389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:48.268250 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:48.269152 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:48.270508 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:48.270899 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:48.271019 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:48.273103 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:48.273109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:48.273114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:48.273142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:48.275061 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:48.275718 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:48.275722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:48.275724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:48.275833 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:48.276237 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:48.276241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:48.276243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:48.276480 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:48.276653 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:48.276985 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 00:55:48.277097 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:55:48.280359 32511 catalog_manager.cc:1347] Generated new cluster ID: 279a9c9dab534f9593ce2eac31dcd700
I20260929 00:55:48.280436 32511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:48.302107 32511 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:48.303067 32511 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:48.309540 32511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e7bd613b90496b8ee40db11a5a6c0e: Generated new TSK 0
I20260929 00:55:48.310364 32511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:48.322212 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32429
W20260929 00:55:48.330961 32414 connection.cc:535] client connection to 127.11.40.254:33897 recv error: Network error: failed to read from TLS socket (remote: 127.11.40.254:33897): Connection reset by peer (error 104)
2026-09-29T00:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32528](info): set up 2 sockets
Sep 29 00:55:48 dist-test-slave-5309 krb5kdc[32528](info): commencing operation
krb5kdc: starting...
W20260929 00:55:49.370523 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:49 dist-test-slave-5309 krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:49.501186 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:45497
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:49.613924 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:49.614326 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:49.614414 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:49.618568 32541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:49.618685 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:49.618726 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:49.618768 32541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:49.618803 32541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:49.624406 32541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:45497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:49.625681 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:49.626778 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:49.636371 32549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:49.636350 32547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:49.636349 32546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:49.637305 32541 server_base.cc:1034] running on GCE node
I20260929 00:55:49.637989 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:49.639292 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:49.640507 32541 hybrid_clock.cc:648] HybridClock initialized: now 1790643349640482 us; error 45 us; skew 500 ppm
Sep 29 00:55:49 dist-test-slave-5309 krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643349, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:49.643766 32541 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:49.644893 32541 webserver.cc:460] Webserver started at http://127.11.40.254:40299/ using document root <none> and password file <none>
I20260929 00:55:49.645509 32541 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:49.645606 32541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:49.645891 32541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:49.648082 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "42536cb4819b44e0ab88f75e8c7eca89"
format_stamp: "Formatted at 2026-09-29 00:55:49 on dist-test-slave-5309"
server_key: "3b3c9a2d18ee52c868221f94ed6902ac"
server_key_iv: "2dec01f40814eadf6aa7c480e114a026"
server_key_version: "encryptionkey@0"
I20260929 00:55:49.648677 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "42536cb4819b44e0ab88f75e8c7eca89"
format_stamp: "Formatted at 2026-09-29 00:55:49 on dist-test-slave-5309"
server_key: "3b3c9a2d18ee52c868221f94ed6902ac"
server_key_iv: "2dec01f40814eadf6aa7c480e114a026"
server_key_version: "encryptionkey@0"
I20260929 00:55:49.652526 32541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:55:49.655085 32556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:49.656407 32541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:49.656569 32541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "42536cb4819b44e0ab88f75e8c7eca89"
format_stamp: "Formatted at 2026-09-29 00:55:49 on dist-test-slave-5309"
server_key: "3b3c9a2d18ee52c868221f94ed6902ac"
server_key_iv: "2dec01f40814eadf6aa7c480e114a026"
server_key_version: "encryptionkey@0"
I20260929 00:55:49.656692 32541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:49.681823 32541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:49.685110 32541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:49.685338 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:49.692936 32541 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:45497
I20260929 00:55:49.692965 32609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:45497 every 8 connection(s)
I20260929 00:55:49.693921 32541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:49.696741 32610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:49.698519 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32541
I20260929 00:55:49.698740 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:49.699038 11427 external_mini_cluster.cc:1428] Setting key 1116b00732c478e2420835bec7432886
I20260929 00:55:49.704104 32610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89: Bootstrap starting.
I20260929 00:55:49.706835 32610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:49.707576 32610 log.cc:824] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:49.709833 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:49.700622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42163 (local address 127.11.40.254:45497)
0929 00:55:49.701106 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:49.701125 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:49.701169 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:55:49.702149 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:49.702178 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:49.702644 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:49.703024 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:49.703496 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:49.704957 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:49.705861 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:49.706537 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:49.706721 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:49.707743 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:49.707749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:49.707765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:49.708290 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:49.708342 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:49.708365 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:49.708512 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:49.708712 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:55:49.708881 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":171,"threads_started":1}
I20260929 00:55:49.709996 32610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89: No bootstrap required, opened a new log
I20260929 00:55:49.713032 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:49.714164 32610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } }
I20260929 00:55:49.714468 32610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:49.714603 32610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42536cb4819b44e0ab88f75e8c7eca89, State: Initialized, Role: FOLLOWER
I20260929 00:55:49.715071 32610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } }
I20260929 00:55:49.715231 32610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:49.715335 32610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:49.715443 32610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:49.716470 32610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } }
I20260929 00:55:49.716898 32610 leader_election.cc:302] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42536cb4819b44e0ab88f75e8c7eca89; no voters:
I20260929 00:55:49.717308 32610 leader_election.cc:288] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:49.717381 32620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:49.717602 32620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [term 1 LEADER]: Becoming Leader. State: Replica: 42536cb4819b44e0ab88f75e8c7eca89, State: Running, Role: LEADER
I20260929 00:55:49.717958 32620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } }
I20260929 00:55:49.718708 32610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:49.720144 32621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42536cb4819b44e0ab88f75e8c7eca89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } } }
I20260929 00:55:49.720567 32621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:49.720443 32622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42536cb4819b44e0ab88f75e8c7eca89. Latest consensus state: current_term: 1 leader_uuid: "42536cb4819b44e0ab88f75e8c7eca89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42536cb4819b44e0ab88f75e8c7eca89" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 45497 } } }
I20260929 00:55:49.720975 32622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:49.721021 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:49.715503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47049 (local address 127.11.40.254:45497)
0929 00:55:49.715647 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:49.715653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:49.715906 (+ 253us) server_negotiation.cc:407] Connection header received
0929 00:55:49.716084 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:49.716087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:49.716154 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:49.716252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:49.716557 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:49.717287 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:49.718447 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:49.718845 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:49.718957 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:49.720004 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:49.720011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:49.720015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:49.720034 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:49.720072 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:49.720091 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:49.720190 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:49.720783 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 00:55:49.720857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:55:49.721184 32634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:49.723886 32634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:49.730301 32634 catalog_manager.cc:1347] Generated new cluster ID: 9542734991bb4d48b4082b03805e5f59
I20260929 00:55:49.730386 32634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:49.750355 32634 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:49.751550 32634 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:49.758298 32634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42536cb4819b44e0ab88f75e8c7eca89: Generated new TSK 0
I20260929 00:55:49.759022 32634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:49.764555 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32541
W20260929 00:55:49.770267 32526 connection.cc:535] client connection to 127.11.40.254:45497 recv error: Network error: failed to read from TLS socket (remote: 127.11.40.254:45497): Connection reset by peer (error 104)
2026-09-29T00:55:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:49 dist-test-slave-5309 krb5kdc[32641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:49 dist-test-slave-5309 krb5kdc[32641](info): set up 2 sockets
Sep 29 00:55:49 dist-test-slave-5309 krb5kdc[32641](info): commencing operation
krb5kdc: starting...
W20260929 00:55:50.822574 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:50 dist-test-slave-5309 krb5kdc[32641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:50.948944 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41431
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:51.069132 32654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:51.069459 32654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:51.069509 32654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:51.074177 32654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:51.074254 32654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:51.074272 32654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:51.074290 32654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:51.074342 32654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:51.079967 32654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:41431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:51.081240 32654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:51.082422 32654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:51.091725 32662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:51.091722 32660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:51.091719 32659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:51.091940 32654 server_base.cc:1034] running on GCE node
I20260929 00:55:51.092537 32654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:51.093825 32654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:51.095037 32654 hybrid_clock.cc:648] HybridClock initialized: now 1790643351094986 us; error 60 us; skew 500 ppm
Sep 29 00:55:51 dist-test-slave-5309 krb5kdc[32641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643351, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:51.098096 32654 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:51.098954 32654 webserver.cc:460] Webserver started at http://127.11.40.254:39371/ using document root <none> and password file <none>
I20260929 00:55:51.099480 32654 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:51.099538 32654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:51.099706 32654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:51.101796 32654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8"
format_stamp: "Formatted at 2026-09-29 00:55:51 on dist-test-slave-5309"
server_key: "28f1c7486c7705e684b26ccf9998cd2b"
server_key_iv: "651f84bf0bac54ae9553537ece814a80"
server_key_version: "encryptionkey@0"
I20260929 00:55:51.102337 32654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8"
format_stamp: "Formatted at 2026-09-29 00:55:51 on dist-test-slave-5309"
server_key: "28f1c7486c7705e684b26ccf9998cd2b"
server_key_iv: "651f84bf0bac54ae9553537ece814a80"
server_key_version: "encryptionkey@0"
I20260929 00:55:51.106053 32654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 00:55:51.108525 32669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:51.109926 32654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:51.110080 32654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8"
format_stamp: "Formatted at 2026-09-29 00:55:51 on dist-test-slave-5309"
server_key: "28f1c7486c7705e684b26ccf9998cd2b"
server_key_iv: "651f84bf0bac54ae9553537ece814a80"
server_key_version: "encryptionkey@0"
I20260929 00:55:51.110199 32654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:51.114908 32654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:51.118288 32654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:51.118515 32654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:51.125734 32654 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:41431
I20260929 00:55:51.125830 32721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:41431 every 8 connection(s)
I20260929 00:55:51.126780 32654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:51.129819 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:51.135543 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8: Bootstrap starting.
I20260929 00:55:51.136104 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32654
I20260929 00:55:51.136251 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:51.136538 11427 external_mini_cluster.cc:1428] Setting key 02dbed62465d2fccae9846e5b3b2e701
I20260929 00:55:51.138693 32722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:51.139482 32722 log.cc:824] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:51.141847 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8: No bootstrap required, opened a new log
I20260929 00:55:51.142216 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:51.137994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51093 (local address 127.11.40.254:41431)
0929 00:55:51.138476 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:51.138491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:51.138527 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:55:51.139312 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:51.139338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:51.139683 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:51.140023 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:51.140756 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:51.140760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:51.140771 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:51.140779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:51.140903 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:51.140926 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:51.141017 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:51.141174 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 00:55:51.141400 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":149,"threads_started":1}
I20260929 00:55:51.145320 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:51.146575 32722 raft_consensus.cc:348] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } }
I20260929 00:55:51.146930 32722 raft_consensus.cc:374] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:51.147012 32722 raft_consensus.cc:729] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f987cc8fd1eb4813a4ce0ba017f7d0f8, State: Initialized, Role: FOLLOWER
I20260929 00:55:51.147543 32722 consensus_queue.cc:260] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } }
I20260929 00:55:51.147684 32722 raft_consensus.cc:388] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:51.147794 32722 raft_consensus.cc:482] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:51.147926 32722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:51.149380 32722 raft_consensus.cc:504] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } }
I20260929 00:55:51.149792 32722 leader_election.cc:302] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f987cc8fd1eb4813a4ce0ba017f7d0f8; no voters:
I20260929 00:55:51.150059 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:51.147880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34805 (local address 127.11.40.254:41431)
0929 00:55:51.148029 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:51.148034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:51.148369 (+ 335us) server_negotiation.cc:407] Connection header received
0929 00:55:51.148513 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:51.148515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:51.148551 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:51.148614 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:51.149496 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:51.149499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:51.149567 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:51.149585 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:51.149622 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:51.149652 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:51.149748 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:51.149871 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:55:51.149929 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:55:51.150178 32722 leader_election.cc:288] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:51.150244 32732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:51.150465 32732 raft_consensus.cc:686] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [term 1 LEADER]: Becoming Leader. State: Replica: f987cc8fd1eb4813a4ce0ba017f7d0f8, State: Running, Role: LEADER
I20260929 00:55:51.150776 32732 consensus_queue.cc:237] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } }
I20260929 00:55:51.151578 32722 sys_catalog.cc:564] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:51.152022 32733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f987cc8fd1eb4813a4ce0ba017f7d0f8. Latest consensus state: current_term: 1 leader_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } } }
I20260929 00:55:51.152050 32734 sys_catalog.cc:455] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f987cc8fd1eb4813a4ce0ba017f7d0f8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 41431 } } }
I20260929 00:55:51.152166 32734 sys_catalog.cc:458] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:51.152165 32733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:51.152613 32737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:51.155525 32737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:51.161652 32737 catalog_manager.cc:1347] Generated new cluster ID: 0faf2c857b014c39848f3f3ad2238b0f
I20260929 00:55:51.161800 32737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:51.175580 32737 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:51.175911 32737 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:51.179914 32737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f987cc8fd1eb4813a4ce0ba017f7d0f8: Generated new TSK 0
I20260929 00:55:51.180605 32737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:51.186892 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32654
2026-09-29T00:55:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:51 dist-test-slave-5309 krb5kdc[32753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:51 dist-test-slave-5309 krb5kdc[32753](info): set up 2 sockets
Sep 29 00:55:51 dist-test-slave-5309 krb5kdc[32753](info): commencing operation
krb5kdc: starting...
W20260929 00:55:52.250524 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:52.381296 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:46029
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:52.497514 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:52.497822 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:52.497866 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:52.502108 32766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:52.502183 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:52.502204 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:52.502221 32766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:52.502236 32766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:52.507753 32766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:52.508997 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:52.510051 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:52.519148 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:52.519182 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:52.519358 32766 server_base.cc:1034] running on GCE node
W20260929 00:55:52.519199 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:52.520154 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:52.521440 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:52.522660 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790643352522647 us; error 30 us; skew 500 ppm
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643352, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:52.525686 32766 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:52.526600 32766 webserver.cc:460] Webserver started at http://127.11.40.254:40337/ using document root <none> and password file <none>
I20260929 00:55:52.527127 32766 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:52.527185 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:52.527356 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:52.529510 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d"
format_stamp: "Formatted at 2026-09-29 00:55:52 on dist-test-slave-5309"
server_key: "d1eacd04ff37b0b6a42e82da7fdd87c8"
server_key_iv: "b05ae22ccb33191a93b9e4680107d5d2"
server_key_version: "encryptionkey@0"
I20260929 00:55:52.530031 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d"
format_stamp: "Formatted at 2026-09-29 00:55:52 on dist-test-slave-5309"
server_key: "d1eacd04ff37b0b6a42e82da7fdd87c8"
server_key_iv: "b05ae22ccb33191a93b9e4680107d5d2"
server_key_version: "encryptionkey@0"
I20260929 00:55:52.533815 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:55:52.536448 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:52.537861 32766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:52.537976 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d"
format_stamp: "Formatted at 2026-09-29 00:55:52 on dist-test-slave-5309"
server_key: "d1eacd04ff37b0b6a42e82da7fdd87c8"
server_key_iv: "b05ae22ccb33191a93b9e4680107d5d2"
server_key_version: "encryptionkey@0"
I20260929 00:55:52.538071 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:52.567126 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:52.570963 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:52.571141 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:52.579905 32766 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:46029
I20260929 00:55:52.579923 365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:46029 every 8 connection(s)
I20260929 00:55:52.581063 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:52.584383 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:52.589133 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 32766
I20260929 00:55:52.589262 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:52.589563 11427 external_mini_cluster.cc:1428] Setting key fbc0e72ed51d9a9c8e04a8f055f7ade2
I20260929 00:55:52.592185 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d: Bootstrap starting.
I20260929 00:55:52.594820 366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:52.595621 366 log.cc:824] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:52.597797 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d: No bootstrap required, opened a new log
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[32753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:52.601394 366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } }
I20260929 00:55:52.601651 366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:52.601732 366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6ddea5ab164b6fa00469ecfeda1c5d, State: Initialized, Role: FOLLOWER
I20260929 00:55:52.602242 366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } }
I20260929 00:55:52.602397 366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:52.602483 366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:52.602581 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:52.603562 366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } }
I20260929 00:55:52.603904 366 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a6ddea5ab164b6fa00469ecfeda1c5d; no voters:
I20260929 00:55:52.604261 366 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:52.604353 371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:52.604568 371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [term 1 LEADER]: Becoming Leader. State: Replica: 2a6ddea5ab164b6fa00469ecfeda1c5d, State: Running, Role: LEADER
I20260929 00:55:52.604575 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:52.591123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44971 (local address 127.11.40.254:46029)
0929 00:55:52.591648 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:52.591664 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:52.591701 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:55:52.592445 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:52.592465 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:52.592892 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:52.593240 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:52.594090 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:52.595209 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:52.596161 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:52.596684 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:52.596913 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:52.599476 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:52.599481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:52.599495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:52.599522 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:52.601365 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:52.601855 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:52.601860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:52.601865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:52.601977 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:52.602276 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:52.602279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:52.602281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:52.602837 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:52.603052 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:52.603338 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 00:55:52.603635 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":169,"threads_started":1}
I20260929 00:55:52.604934 371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } }
I20260929 00:55:52.606736 372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6ddea5ab164b6fa00469ecfeda1c5d. Latest consensus state: current_term: 1 leader_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } } }
I20260929 00:55:52.606777 373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6ddea5ab164b6fa00469ecfeda1c5d" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 46029 } } }
I20260929 00:55:52.606956 373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:52.606984 372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:52.607282 366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:52.607411 375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:52.607896 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:52.610929 375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:52.617028 375 catalog_manager.cc:1347] Generated new cluster ID: 096ad523d9784ee88e3607cd5193d22f
I20260929 00:55:52.617095 375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:52.619858 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:52.610268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43065 (local address 127.11.40.254:46029)
0929 00:55:52.610377 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:52.610382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:52.610659 (+ 277us) server_negotiation.cc:407] Connection header received
0929 00:55:52.610790 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:52.610793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:52.610867 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:52.610956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:52.612106 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:52.612959 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:52.614060 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:52.614452 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:52.614565 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:52.616276 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:52.616280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:52.616283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:52.616309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:52.618021 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:52.618546 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:52.618549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:52.618551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:52.618645 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:52.618933 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:52.618940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:52.618941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:52.619172 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:52.619304 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:52.619556 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 00:55:52.619682 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:55:52.633594 375 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:52.634577 375 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:52.641085 375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a6ddea5ab164b6fa00469ecfeda1c5d: Generated new TSK 0
I20260929 00:55:52.641916 375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:52.651151 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 32766
2026-09-29T00:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[397](info): set up 2 sockets
Sep 29 00:55:52 dist-test-slave-5309 krb5kdc[397](info): commencing operation
krb5kdc: starting...
W20260929 00:55:53.727115 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:53 dist-test-slave-5309 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:53.861970 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37217
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:37217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:53.977391 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:53.977694 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:53.977739 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:53.981925 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:53.982008 410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:53.982028 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:53.982046 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:53.982120 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:53.987751 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:37217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:53.989364 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:53.990583 410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:54.000226 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:54.000344 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:54.000226 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:54.000597 410 server_base.cc:1034] running on GCE node
I20260929 00:55:54.001325 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:54.002563 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:54.003779 410 hybrid_clock.cc:648] HybridClock initialized: now 1790643354003749 us; error 49 us; skew 500 ppm
Sep 29 00:55:54 dist-test-slave-5309 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643354, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:54.013396 410 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:54.014480 410 webserver.cc:460] Webserver started at http://127.11.40.254:39113/ using document root <none> and password file <none>
I20260929 00:55:54.015091 410 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:54.015192 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:54.015437 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:54.017640 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "e8826e8cc52748b2a8a3adf96b5aaa27"
format_stamp: "Formatted at 2026-09-29 00:55:54 on dist-test-slave-5309"
server_key: "8dd48526097a491259f939955210280b"
server_key_iv: "998d1b032caeac448b83647d212f6ceb"
server_key_version: "encryptionkey@0"
I20260929 00:55:54.018241 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "e8826e8cc52748b2a8a3adf96b5aaa27"
format_stamp: "Formatted at 2026-09-29 00:55:54 on dist-test-slave-5309"
server_key: "8dd48526097a491259f939955210280b"
server_key_iv: "998d1b032caeac448b83647d212f6ceb"
server_key_version: "encryptionkey@0"
I20260929 00:55:54.022368 410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:55:54.025151 425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:54.026517 410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:54.026676 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "e8826e8cc52748b2a8a3adf96b5aaa27"
format_stamp: "Formatted at 2026-09-29 00:55:54 on dist-test-slave-5309"
server_key: "8dd48526097a491259f939955210280b"
server_key_iv: "998d1b032caeac448b83647d212f6ceb"
server_key_version: "encryptionkey@0"
I20260929 00:55:54.026801 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:54.037954 410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:54.041183 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:54.041409 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:54.048488 410 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:37217
I20260929 00:55:54.048516 477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:37217 every 8 connection(s)
I20260929 00:55:54.049525 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:54.052229 478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:54.057981 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27: Bootstrap starting.
I20260929 00:55:54.059338 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 410
I20260929 00:55:54.059476 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:54.059782 11427 external_mini_cluster.cc:1428] Setting key a7feaf0c2350633873d313bf783a0221
I20260929 00:55:54.060901 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:54.061872 478 log.cc:824] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:54.064240 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27: No bootstrap required, opened a new log
I20260929 00:55:54.068969 478 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } }
I20260929 00:55:54.069361 478 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:54.069428 478 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8826e8cc52748b2a8a3adf96b5aaa27, State: Initialized, Role: FOLLOWER
I20260929 00:55:54.069927 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } }
I20260929 00:55:54.070083 478 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:54.070154 478 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:54.070250 478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:54.070391 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:54.061063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37149 (local address 127.11.40.254:37217)
0929 00:55:54.061599 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:54.061620 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:54.061669 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:55:54.062521 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:54.062541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:54.062983 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:54.063368 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:54.063771 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:54.065240 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:54.066161 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:54.066914 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:54.067106 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:54.068132 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:54.068139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:54.068159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:54.068719 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:54.068793 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:54.068855 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:54.068956 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:54.069124 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 00:55:54.069361 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":188,"threads_started":1}
I20260929 00:55:54.071360 478 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } }
I20260929 00:55:54.071727 478 leader_election.cc:302] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8826e8cc52748b2a8a3adf96b5aaa27; no voters:
I20260929 00:55:54.072078 478 leader_election.cc:288] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:54.072199 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:54.072468 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [term 1 LEADER]: Becoming Leader. State: Replica: e8826e8cc52748b2a8a3adf96b5aaa27, State: Running, Role: LEADER
I20260929 00:55:54.072873 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } }
I20260929 00:55:54.073449 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:54.073633 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:54.074931 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8826e8cc52748b2a8a3adf96b5aaa27. Latest consensus state: current_term: 1 leader_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } } }
I20260929 00:55:54.075068 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:54.075408 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826e8cc52748b2a8a3adf96b5aaa27" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 37217 } } }
I20260929 00:55:54.075497 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:54.076336 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:54.079698 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:54.080440 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:54.076263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37081 (local address 127.11.40.254:37217)
0929 00:55:54.076530 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:54.076536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:54.076707 (+ 171us) server_negotiation.cc:407] Connection header received
0929 00:55:54.076955 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:54.076958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:54.077023 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:54.077117 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:54.077402 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:54.077937 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:54.078763 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:54.079013 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:54.079129 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:54.079996 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:54.080000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:54.080002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:54.080015 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:54.080049 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:54.080060 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:54.080141 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:54.080246 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 00:55:54.080298 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260929 00:55:54.085675 495 catalog_manager.cc:1347] Generated new cluster ID: c71a22bb23e54c0ea623fee086379104
I20260929 00:55:54.085809 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:54.111259 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:54.113018 495 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:54.122649 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8826e8cc52748b2a8a3adf96b5aaa27: Generated new TSK 0
I20260929 00:55:54.123518 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:54.137627 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 410
2026-09-29T00:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:54 dist-test-slave-5309 krb5kdc[509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:54 dist-test-slave-5309 krb5kdc[509](info): set up 2 sockets
Sep 29 00:55:54 dist-test-slave-5309 krb5kdc[509](info): commencing operation
krb5kdc: starting...
W20260929 00:55:55.194528 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:55 dist-test-slave-5309 krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:55.320478 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39191
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:39191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:55.430927 522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:55.431187 522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:55.431231 522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:55.435338 522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:55.435397 522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:55.435417 522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:55.435433 522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:55.435448 522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:55.440716 522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:39191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:39191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:55.441856 522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:55.442884 522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:55.451941 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:55.451992 530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:55.451995 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:55.452366 522 server_base.cc:1034] running on GCE node
I20260929 00:55:55.453047 522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:55.454386 522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:55.455657 522 hybrid_clock.cc:648] HybridClock initialized: now 1790643355455703 us; error 93 us; skew 500 ppm
Sep 29 00:55:55 dist-test-slave-5309 krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643355, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:55.458704 522 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:55.459630 522 webserver.cc:460] Webserver started at http://127.11.40.254:42605/ using document root <none> and password file <none>
I20260929 00:55:55.460237 522 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:55.460337 522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:55.460613 522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:55.462838 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "3059cff107004d09af00c39f34a451d8"
format_stamp: "Formatted at 2026-09-29 00:55:55 on dist-test-slave-5309"
server_key: "26047cb41154966db296d8c4bb28b9f6"
server_key_iv: "28bf1edca67eec0db62af50005b657b6"
server_key_version: "encryptionkey@0"
I20260929 00:55:55.463451 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "3059cff107004d09af00c39f34a451d8"
format_stamp: "Formatted at 2026-09-29 00:55:55 on dist-test-slave-5309"
server_key: "26047cb41154966db296d8c4bb28b9f6"
server_key_iv: "28bf1edca67eec0db62af50005b657b6"
server_key_version: "encryptionkey@0"
I20260929 00:55:55.467355 522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 00:55:55.469843 537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:55.471114 522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:55:55.471369 522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "3059cff107004d09af00c39f34a451d8"
format_stamp: "Formatted at 2026-09-29 00:55:55 on dist-test-slave-5309"
server_key: "26047cb41154966db296d8c4bb28b9f6"
server_key_iv: "28bf1edca67eec0db62af50005b657b6"
server_key_version: "encryptionkey@0"
I20260929 00:55:55.471488 522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:55.476269 522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:55.479837 522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:55.480065 522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:55.487143 522 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:39191
I20260929 00:55:55.487197 589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:39191 every 8 connection(s)
I20260929 00:55:55.488214 522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:55.491139 590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:55.496750 590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8: Bootstrap starting.
I20260929 00:55:55.497366 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 522
I20260929 00:55:55.497552 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:55.497834 11427 external_mini_cluster.cc:1428] Setting key 0c2e569e3b7ebc4798bcf2ee910293dc
I20260929 00:55:55.499655 590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:55.500423 590 log.cc:824] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:55.502724 590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8: No bootstrap required, opened a new log
I20260929 00:55:55.503279 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:55.499159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44267 (local address 127.11.40.254:39191)
0929 00:55:55.499608 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:55.499623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:55.499658 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:55:55.500352 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:55.500370 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:55.500703 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:55.501106 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:55.501912 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:55.501916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:55.501929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:55.501937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:55.501987 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:55.502004 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:55.502091 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:55.502279 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 00:55:55.502480 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":134,"threads_started":1}
I20260929 00:55:55.506421 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:55.507010 590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } }
I20260929 00:55:55.507313 590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:55.507385 590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3059cff107004d09af00c39f34a451d8, State: Initialized, Role: FOLLOWER
I20260929 00:55:55.507911 590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } }
I20260929 00:55:55.508098 590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:55.508174 590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:55.508275 590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:55.509584 590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } }
I20260929 00:55:55.510021 590 leader_election.cc:302] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3059cff107004d09af00c39f34a451d8; no voters:
I20260929 00:55:55.510371 590 leader_election.cc:288] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:55.510470 600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:55.510748 600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3059cff107004d09af00c39f34a451d8, State: Running, Role: LEADER
I20260929 00:55:55.511097 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:55.508934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37557 (local address 127.11.40.254:39191)
0929 00:55:55.509070 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:55.509075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:55.509385 (+ 310us) server_negotiation.cc:407] Connection header received
0929 00:55:55.509505 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:55.509507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:55.509541 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:55.509650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:55.510576 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:55.510581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:55.510586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:55.510599 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:55.510632 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:55.510648 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:55.510760 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:55.510937 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 00:55:55.510983 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:55:55.511240 600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } }
I20260929 00:55:55.511807 590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:55.513108 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3059cff107004d09af00c39f34a451d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } } }
I20260929 00:55:55.513248 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:55.513615 602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3059cff107004d09af00c39f34a451d8. Latest consensus state: current_term: 1 leader_uuid: "3059cff107004d09af00c39f34a451d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3059cff107004d09af00c39f34a451d8" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 39191 } } }
I20260929 00:55:55.513710 602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:55.514189 609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:55.517110 609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:55.522403 609 catalog_manager.cc:1347] Generated new cluster ID: 260aabbc16934e9db7fe8788c05d8906
I20260929 00:55:55.522471 609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:55.542891 609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:55.543246 609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:55.553169 609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3059cff107004d09af00c39f34a451d8: Generated new TSK 0
I20260929 00:55:55.554044 609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:55.564163 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 522
2026-09-29T00:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1426 ms)
[----------] 18 tests from AuthTokenIssuingTest (26365 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:55 dist-test-slave-5309 krb5kdc[621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:55 dist-test-slave-5309 krb5kdc[621](info): set up 2 sockets
Sep 29 00:55:55 dist-test-slave-5309 krb5kdc[621](info): commencing operation
krb5kdc: starting...
W20260929 00:55:56.622574 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:56 dist-test-slave-5309 krb5kdc[621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:56.755908 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33855
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:56.871227 634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:55:56.871649 634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:56.871726 634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:56.871801 634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:56.876080 634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:56.876156 634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:56.876176 634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:56.876194 634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:56.876209 634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:56.882143 634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:56.883365 634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:56.884433 634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:56.893174 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:56.893311 634 server_base.cc:1034] running on GCE node
W20260929 00:55:56.893150 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:56.893146 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:56.894047 634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:56.895231 634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:56.896438 634 hybrid_clock.cc:648] HybridClock initialized: now 1790643356896409 us; error 42 us; skew 500 ppm
Sep 29 00:55:56 dist-test-slave-5309 krb5kdc[621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:56.899547 634 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:55:56.900570 634 webserver.cc:460] Webserver started at http://127.11.40.254:42015/ using document root <none> and password file <none>
I20260929 00:55:56.901224 634 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:56.901322 634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:56.901567 634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:56.903712 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "b3530ba2be424806b4d1767e7f4333e4"
format_stamp: "Formatted at 2026-09-29 00:55:56 on dist-test-slave-5309"
server_key: "5d8f64513d3b96133231c8020717ee2e"
server_key_iv: "0001e02c36fec71d9eda5811e0ae7522"
server_key_version: "encryptionkey@0"
I20260929 00:55:56.904357 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "b3530ba2be424806b4d1767e7f4333e4"
format_stamp: "Formatted at 2026-09-29 00:55:56 on dist-test-slave-5309"
server_key: "5d8f64513d3b96133231c8020717ee2e"
server_key_iv: "0001e02c36fec71d9eda5811e0ae7522"
server_key_version: "encryptionkey@0"
I20260929 00:55:56.908324 634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:55:56.911195 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:56.912549 634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:56.912652 634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "b3530ba2be424806b4d1767e7f4333e4"
format_stamp: "Formatted at 2026-09-29 00:55:56 on dist-test-slave-5309"
server_key: "5d8f64513d3b96133231c8020717ee2e"
server_key_iv: "0001e02c36fec71d9eda5811e0ae7522"
server_key_version: "encryptionkey@0"
I20260929 00:55:56.912743 634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:56.935647 634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:56.938840 634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:56.939011 634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:56.946681 634 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:33855
I20260929 00:55:56.947002 701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:33855 every 8 connection(s)
I20260929 00:55:56.947726 634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:56.950721 702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:56.953109 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 634
I20260929 00:55:56.953212 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:55:56.953495 11427 external_mini_cluster.cc:1428] Setting key 77a54e7b1711bc39181be2282d3dc404
I20260929 00:55:56.954723 702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.253@KRBTEST.COM" created.
Sep 29 00:55:56 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:56.971377 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:56.957206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.11.40.254:33855)
0929 00:55:56.957790 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:56.957810 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:56.958078 (+ 268us) server_negotiation.cc:407] Connection header received
0929 00:55:56.959673 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:56.959678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:56.959858 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:56.960009 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:56.961175 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:56.962087 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:56.963206 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:56.963595 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:56.963714 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:56.966234 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:56.966241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:56.966259 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:56.966280 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:56.968016 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:56.968541 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:56.968547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:56.968548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:56.968649 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:56.969024 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:56.969027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:56.969029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:56.969421 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:56.969570 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:56.969926 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 00:55:56.970263 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":446,"thread_start_us":140,"threads_started":1}
I20260929 00:55:56.971377 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:56.957206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33855 (local address 127.0.0.1:59568)
0929 00:55:56.957885 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:56.957946 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:56.959318 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:56.960074 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:56.960080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:56.960588 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:56.961002 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:56.961026 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:56.962237 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:56.962243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:56.963104 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:56.963113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:56.963738 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:56.963741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:56.963829 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:56.964453 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:56.964470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:56.966083 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:56.968126 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:56.968134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:56.968143 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:56.968410 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:56.968744 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:56.968749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:56.968751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:56.968924 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:56.969540 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:56.969546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:56.969687 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 00:55:56.969853 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 00:55:56.970314 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":164,"spinlock_wait_cycles":34176,"thread_start_us":160,"threads_started":1}
I20260929 00:55:56.973838 702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:56.974680 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:56.974411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58160)
0929 00:55:56.974535 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:56.974614 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:55:56.974880 651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
I20260929 00:55:56.975843 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:56.975605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58174)
0929 00:55:56.975717 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:56.975769 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:55:56.976012 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:36723: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
I20260929 00:55:56.985111 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:36723
--webserver_interface=127.11.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:55:57.025242 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } attempt: 1
I20260929 00:55:57.026202 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.025899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58190)
0929 00:55:57.026060 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.026124 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 00:55:57.026978 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.026741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58204)
0929 00:55:57.026880 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.026923 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:55:57.027107 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:36723: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
I20260929 00:55:57.095279 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } attempt: 2
I20260929 00:55:57.096205 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.095868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58210)
0929 00:55:57.096048 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.096113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 00:55:57.098518 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.098254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58224)
0929 00:55:57.098382 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.098444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:55:57.098701 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:36723: Network error: Client connection negotiation failed: client connection to 127.11.40.253:36723: connect: Connection refused (error 111)
W20260929 00:55:57.100258 709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:55:57.100606 709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:57.100663 709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:57.100728 709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:57.104987 709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:57.105044 709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:57.105064 709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:57.105082 709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:57.105098 709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:57.110635 709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:36723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:57.111909 709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:57.112973 709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:57.121663 717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:57.121652 715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:57.121883 709 server_base.cc:1034] running on GCE node
W20260929 00:55:57.121676 714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:57.122735 709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:57.123972 709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:57.125267 709 hybrid_clock.cc:648] HybridClock initialized: now 1790643357125242 us; error 44 us; skew 500 ppm
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:57.128332 709 init.cc:377] Logged in from keytab as kudu/127.11.40.253@KRBTEST.COM (short username kudu)
I20260929 00:55:57.129288 709 webserver.cc:460] Webserver started at http://127.11.40.253:44851/ using document root <none> and password file <none>
I20260929 00:55:57.129830 709 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:57.129890 709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:57.130061 709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:57.132280 709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data/instance:
uuid: "3532838e91c74538b17332ce30bb0862"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "f676008f2ab17d5a70d7d33fed33ada0"
server_key_iv: "083231ccbf2641f6bdc5325df498f0ee"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.132923 709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal/instance:
uuid: "3532838e91c74538b17332ce30bb0862"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "f676008f2ab17d5a70d7d33fed33ada0"
server_key_iv: "083231ccbf2641f6bdc5325df498f0ee"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.136912 709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:55:57.139400 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:57.140941 709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:55:57.141084 709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "3532838e91c74538b17332ce30bb0862"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "f676008f2ab17d5a70d7d33fed33ada0"
server_key_iv: "083231ccbf2641f6bdc5325df498f0ee"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.141235 709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:57.153971 709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:57.157593 709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:57.157778 709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:57.165928 709 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.253:36723
I20260929 00:55:57.166262 776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.253:36723 every 8 connection(s)
I20260929 00:55:57.167018 709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
I20260929 00:55:57.170037 777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:57.172292 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 709
I20260929 00:55:57.172406 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-1/wal/instance
I20260929 00:55:57.172713 11427 external_mini_cluster.cc:1428] Setting key dc5c2aa5009b57705afdf915c719878a
I20260929 00:55:57.174218 777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.40.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.252@KRBTEST.COM" created.
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:57.189893 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } attempt: 3
I20260929 00:55:57.190698 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.176758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.11.40.254:33855)
0929 00:55:57.176977 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.176984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.178028 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 00:55:57.179011 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.179016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.179080 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.179189 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.180396 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.181301 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.182728 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.182973 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.183111 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.186162 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.186165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.186167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.186192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.188338 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.188919 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.188923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.188927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.189010 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.189381 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.189384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.189386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.189574 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.189681 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.190347 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.190506 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 00:55:57.191836 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.177236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33855 (local address 127.0.0.1:59574)
0929 00:55:57.177838 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.177896 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.178783 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.179349 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.179355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.179696 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.180213 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.180227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.181437 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.181440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.182614 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.182624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.183204 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.183208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.183365 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.184083 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.184103 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.185968 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.188471 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.188478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.188496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.188777 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.189119 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.189123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.189125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.189276 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.189694 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.189703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.189977 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.190340 (+ 363us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.190713 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":173,"threads_started":1}
I20260929 00:55:57.193575 777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:55:57.200778 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.190387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58238)
0929 00:55:57.190694 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.190710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.190785 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.191782 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.191787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.191986 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.192105 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.192115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.193236 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.193239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.193937 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.193949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.194761 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.194765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.194888 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.195423 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.195435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.196665 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.199027 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.199029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.199031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.199249 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.199613 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.199616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.199619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.199709 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.200230 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.200233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.200342 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.200462 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.200606 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260929 00:55:57.200942 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.190929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.11.40.253:36723)
0929 00:55:57.191347 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.191358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.191380 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:55:57.191421 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.191426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.191627 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.191764 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.192224 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.193119 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.194090 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.194608 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.194749 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.196854 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.196868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.196886 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.196918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.198917 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.199367 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.199372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.199373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.199512 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.199819 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.199822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.199824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.200120 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.200264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.200578 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.200719 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
I20260929 00:55:57.202477 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:35779
--webserver_interface=127.11.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:55:57.203191 702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:57.203393 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.194115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58242)
0929 00:55:57.194228 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.194240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.194310 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.194870 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.194872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.195002 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.195121 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.195129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.195991 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.195994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.196649 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.196658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.197184 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.197187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.197314 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.197660 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.197669 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.199160 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.201910 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.201913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.201915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.202162 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.202449 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.202452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.202454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.202534 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.202945 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.202948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.203030 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.203128 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.203250 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":20}
I20260929 00:55:57.203872 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.194249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.11.40.253:36723)
0929 00:55:57.194555 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.194566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.194581 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:57.194618 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.194622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.194674 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.194769 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.195235 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.195879 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.196788 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.197078 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.197187 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.199313 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.199320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.199326 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.199353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.201787 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.202262 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.202268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.202271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.202355 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.202654 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.202658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.202661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.202851 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.202956 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.203563 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.203676 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260929 00:55:57.204224 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.203663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:60994)
0929 00:55:57.204121 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.204167 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
I20260929 00:55:57.205044 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.204758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32768)
0929 00:55:57.204931 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.204985 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 00:55:57.205154 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.205152 777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:57.205855 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.205614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32780)
0929 00:55:57.205721 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.205806 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:55:57.206061 727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.207026 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.206812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32790)
0929 00:55:57.206928 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.206972 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 00:55:57.207167 777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.236326 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 1
I20260929 00:55:57.237309 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.236988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32806)
0929 00:55:57.237150 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.237231 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:55:57.238116 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.237856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32822)
0929 00:55:57.237992 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.238052 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 00:55:57.238250 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.256475 777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 1
I20260929 00:55:57.257426 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.257118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32832)
0929 00:55:57.257275 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.257345 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:55:57.258172 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.257959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32842)
0929 00:55:57.258069 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.258120 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:55:57.258334 777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.313441 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 2
I20260929 00:55:57.314337 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.314042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32854)
0929 00:55:57.314192 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.314264 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:55:57.315182 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.314947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32868)
0929 00:55:57.315065 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.315118 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 00:55:57.315322 702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
W20260929 00:55:57.322291 784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:55:57.322633 784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:57.322701 784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:57.322790 784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:55:57.326504 777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 2
W20260929 00:55:57.326925 784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:57.327006 784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:57.327076 784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:57.327118 784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:57.327160 784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:57.327382 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.327029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32884)
0929 00:55:57.327239 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.327307 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 00:55:57.328095 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.327886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32896)
0929 00:55:57.327991 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.328036 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:55:57.328212 777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:35779: Network error: Client connection negotiation failed: client connection to 127.11.40.252:35779: connect: Connection refused (error 111)
I20260929 00:55:57.333236 784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:33855,127.11.40.253:36723,127.11.40.252:35779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:55:57.334380 784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:57.335423 784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:57.344069 790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:57.344074 791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:57.344239 784 server_base.cc:1034] running on GCE node
W20260929 00:55:57.344069 793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:57.345013 784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:57.346140 784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:57.347340 784 hybrid_clock.cc:648] HybridClock initialized: now 1790643357347322 us; error 36 us; skew 500 ppm
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:57.350262 784 init.cc:377] Logged in from keytab as kudu/127.11.40.252@KRBTEST.COM (short username kudu)
I20260929 00:55:57.351215 784 webserver.cc:460] Webserver started at http://127.11.40.252:45215/ using document root <none> and password file <none>
I20260929 00:55:57.351775 784 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:57.351866 784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:57.352106 784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:57.354231 784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data/instance:
uuid: "b9a61ac67c334e98b57711d2587f2ae8"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "4ecb6465db653fe4e42d8cb60cd71576"
server_key_iv: "8e93a36026d1597ee1d9236c5194daa9"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.354796 784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal/instance:
uuid: "b9a61ac67c334e98b57711d2587f2ae8"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "4ecb6465db653fe4e42d8cb60cd71576"
server_key_iv: "8e93a36026d1597ee1d9236c5194daa9"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.358510 784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 00:55:57.361018 800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:57.362318 784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:57.362471 784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "b9a61ac67c334e98b57711d2587f2ae8"
format_stamp: "Formatted at 2026-09-29 00:55:57 on dist-test-slave-5309"
server_key: "4ecb6465db653fe4e42d8cb60cd71576"
server_key_iv: "8e93a36026d1597ee1d9236c5194daa9"
server_key_version: "encryptionkey@0"
I20260929 00:55:57.362586 784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:57.380366 784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:57.383989 784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:57.384212 784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:57.391325 784 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.252:35779
I20260929 00:55:57.391638 852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.252:35779 every 8 connection(s)
I20260929 00:55:57.392372 784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
I20260929 00:55:57.395185 853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:57.398990 853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:57.400204 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 784
I20260929 00:55:57.400341 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790642980693194-11427-0/minicluster-data/master-2/wal/instance
I20260929 00:55:57.400622 11427 external_mini_cluster.cc:1428] Setting key 64e14e4ff14f15cece07a69c26fd3f5c
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:55:57.414021 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.401191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.11.40.254:33855)
0929 00:55:57.401347 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.401351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.402341 (+ 990us) server_negotiation.cc:407] Connection header received
0929 00:55:57.403419 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.403422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.403471 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.403545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.404489 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.405095 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.406261 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.406588 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.406683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.409780 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.409786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.409792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.409833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.411534 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.412163 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.412166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.412167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.412254 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.412634 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.412640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.412642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.412956 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.413081 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.413701 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.413843 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:55:57.414491 702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 3
I20260929 00:55:57.415066 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.401680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.11.40.252:35779)
0929 00:55:57.402527 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.402538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.402573 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:55:57.403186 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.403209 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.403443 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.403640 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.404382 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.405516 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.406477 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.406938 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.407041 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.410279 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.410287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.410303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.410334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.412280 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.412787 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.412791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.412793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.412922 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.413226 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.413229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.413231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.413525 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.413730 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.413964 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.414349 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"spinlock_wait_cycles":2432,"thread_start_us":168,"threads_started":1}
I20260929 00:55:57.415223 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.401671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:33855 (local address 127.0.0.1:59584)
0929 00:55:57.402204 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.402246 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.403186 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.403627 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.403632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.404074 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.404305 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.404343 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.405210 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.405213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.406105 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.406113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.406749 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.406753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.406932 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.407707 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.407726 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.409575 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.411648 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.411655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.411666 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.411988 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.412361 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.412367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.412372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.412532 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.413084 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.413089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.413417 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.413625 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.413991 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"spinlock_wait_cycles":54144,"thread_start_us":168,"threads_started":1}
I20260929 00:55:57.416347 853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:57.419344 777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } attempt: 3
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:55:57.425623 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.414938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32920)
0929 00:55:57.415042 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.415058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.415175 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.415798 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.415801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.415981 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.416077 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.416082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.417506 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.417513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.418170 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.418181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.418701 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.418705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.418793 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.419384 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.419401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.421302 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.423776 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.423782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.423784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.424062 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.424432 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.424438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.424443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.424537 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.425056 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.425060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.425183 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.425279 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.425419 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 00:55:57.426400 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.414984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.11.40.252:35779)
0929 00:55:57.415546 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.415555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.415567 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:55:57.415600 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.415602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.415641 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.415786 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.416211 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.417371 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.418295 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.418579 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.418658 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.421450 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.421454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.421457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.421481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.423640 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.424223 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.424229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.424233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.424327 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.424663 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.424669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.424672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.424922 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.425946 (+ 1024us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.426026 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.426180 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":130,"threads_started":1}
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:55:57.426781 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.416862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:36723 (local address 127.0.0.1:58258)
0929 00:55:57.417000 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.417017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.417080 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.417377 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.417379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.417670 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.417816 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.417824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.418739 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.418743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.419523 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.419534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.420240 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.420243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.420357 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.420888 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.420903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.422432 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.425039 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.425042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.425044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.425354 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.425674 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.425676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.425678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.425773 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.426254 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.426257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.426363 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.426469 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.426585 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:55:57.427573 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.417021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.11.40.253:36723)
0929 00:55:57.417134 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.417138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.417157 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:55:57.417194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.417203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.417243 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.417345 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.417962 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.418594 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.419668 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.420011 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.420093 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.422588 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.422592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.422595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.422625 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.424915 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.425479 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.425484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.425486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.425561 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.425893 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.425898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.425901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.426101 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.426181 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.427039 (+ 858us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.427404 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":2944}
I20260929 00:55:57.430397 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.418104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.11.40.254:33855)
0929 00:55:57.418798 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.418801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.418812 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:55:57.418843 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.418846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.418889 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.418964 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.419869 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.420641 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.421820 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.422152 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.422263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.426147 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.426151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.426155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.426179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.427923 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.428963 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.428966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.428968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.429076 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.429403 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.429409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.429414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.429610 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.429791 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.430141 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.430257 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624}
I20260929 00:55:57.430593 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: Bootstrap starting.
I20260929 00:55:57.430802 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.419759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32926)
0929 00:55:57.420123 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.420138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.420227 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.420472 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.420478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.421095 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.421192 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.421199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.423516 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.423521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.424048 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.424056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.424707 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.424711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.424847 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.425321 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.425335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.427347 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.429246 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.429249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.429251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.429451 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.429712 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.429716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.429718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.429891 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.430263 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.430266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.430371 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.430482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.430634 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":896}
I20260929 00:55:57.431649 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.420016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.11.40.252:35779)
0929 00:55:57.420120 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.420123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.420190 (+ 67us) server_negotiation.cc:407] Connection header received
0929 00:55:57.420296 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.420298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.420340 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.421568 (+ 1228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.421659 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.423400 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.424183 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.424558 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.425059 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.427489 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.427493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.427496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.427518 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.429123 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.429549 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.429551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.429552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.429613 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.430001 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.430004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.430006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.430164 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.430270 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.431434 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.431551 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:55:57.433401 702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:57.434303 702 log.cc:824] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:57.434840 777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: Bootstrap starting.
I20260929 00:55:57.434979 853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:55:57.436790 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: No bootstrap required, opened a new log
I20260929 00:55:57.437546 777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: Neither blocks nor log segments found. Creating new log.
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:55:57.438426 777 log.cc:824] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:57.440569 777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: No bootstrap required, opened a new log
Sep 29 00:55:57 dist-test-slave-5309 krb5kdc[621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:55:57.441468 702 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.441798 702 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3530ba2be424806b4d1767e7f4333e4, State: Initialized, Role: FOLLOWER
I20260929 00:55:57.442206 702 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.443715 702 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:57.443827 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.432412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.11.40.253:36723)
0929 00:55:57.432519 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.432523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.432538 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:57.432657 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.432661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.432704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.432786 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.433614 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.434358 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.435403 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.435767 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.435853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.438927 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.438931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.438934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.438961 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.441090 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.441588 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.441592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.441594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.441715 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.442346 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.442352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.442357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.442923 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.443086 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.443487 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.443618 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:55:57.444674 777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.445115 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.435439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:35779 (local address 127.0.0.1:32932)
0929 00:55:57.435783 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:57.435798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:57.435897 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:57.436216 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:57.436219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:57.436402 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:57.436522 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.436532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.437630 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.437633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.438486 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:57.438497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.439194 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.439200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:57.439291 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.439782 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:57.439797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:57.441453 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:57.443423 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.443426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.443428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.443669 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.444002 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:57.444005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:57.444006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:57.444087 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:57.444539 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:57.444542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:57.444627 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 00:55:57.444732 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:55:57.444948 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 00:55:57.445283 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.435680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.11.40.252:35779)
0929 00:55:57.435787 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.435791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.435844 (+ 53us) server_negotiation.cc:407] Connection header received
0929 00:55:57.435991 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.435994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.436034 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.436078 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.436646 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.437504 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.438617 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.438958 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.439032 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.441590 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.441593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.441596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.441617 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.443310 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.443768 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.443770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.443772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.443893 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.444196 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.444200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.444203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.444413 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.444481 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.444778 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.445069 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":31}
I20260929 00:55:57.444960 777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3532838e91c74538b17332ce30bb0862, State: Initialized, Role: FOLLOWER
I20260929 00:55:57.448372 853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: Bootstrap starting.
I20260929 00:55:57.451126 853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:57.447237 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.451581 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.451735 777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.453274 777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:57.453985 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:55:57.455494 676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b3530ba2be424806b4d1767e7f4333e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59598
I20260929 00:55:57.455729 676 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 00:55:57.455847 676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 00:55:57.449512 872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:55:57.456792 872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:55:57.457365 676 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.454571 634 master_runner.cc:186] Error getting master registration for 127.11.40.253:36723: OK, instance_id { permanent_uuid: "3532838e91c74538b17332ce30bb0862" instance_seqno: 1790643357163468 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:55:57.458026 634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:55:57.458477 676 leader_election.cc:288] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [CANDIDATE]: Term 1 election: Requested vote from peers 3532838e91c74538b17332ce30bb0862 (127.11.40.253:36723), b9a61ac67c334e98b57711d2587f2ae8 (127.11.40.252:35779)
I20260929 00:55:57.459260 751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3530ba2be424806b4d1767e7f4333e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3532838e91c74538b17332ce30bb0862"
I20260929 00:55:57.459519 751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:57.457579 853 log.cc:824] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:57.461037 751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3530ba2be424806b4d1767e7f4333e4 in term 1.
I20260929 00:55:57.461464 651 leader_election.cc:302] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3532838e91c74538b17332ce30bb0862, b3530ba2be424806b4d1767e7f4333e4; no voters:
I20260929 00:55:57.461385 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.461758 861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:57.461957 861 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [term 1 LEADER]: Becoming Leader. State: Replica: b3530ba2be424806b4d1767e7f4333e4, State: Running, Role: LEADER
I20260929 00:55:57.461967 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.462289 861 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.464071 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3530ba2be424806b4d1767e7f4333e4. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.464175 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:57.464679 894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:57.465672 894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:57.466778 853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: No bootstrap required, opened a new log
I20260929 00:55:57.463622 709 master_runner.cc:186] Error getting master registration for 127.11.40.252:35779: OK, instance_id { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" instance_seqno: 1790643357389357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:55:57.466964 709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 00:55:57.467298 885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:55:57.467365 885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:55:57.470944 853 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.471206 853 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a61ac67c334e98b57711d2587f2ae8, State: Initialized, Role: FOLLOWER
I20260929 00:55:57.471546 853 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } }
I20260929 00:55:57.473013 853 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:57.473068 827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3530ba2be424806b4d1767e7f4333e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9a61ac67c334e98b57711d2587f2ae8"
I20260929 00:55:57.473320 827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:57.474706 827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3530ba2be424806b4d1767e7f4333e4 in term 1.
I20260929 00:55:57.477023 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.477228 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 00:55:57.479030 910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:55:57.479106 910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:55:57.485239 751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [term 1 FOLLOWER]: Refusing update from remote peer b3530ba2be424806b4d1767e7f4333e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:57.486414 827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [term 1 FOLLOWER]: Refusing update from remote peer b3530ba2be424806b4d1767e7f4333e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:57.486938 861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:57.487416 861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:57.491299 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.469489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.11.40.254:33855)
0929 00:55:57.469602 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.469606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.477894 (+ 8288us) server_negotiation.cc:407] Connection header received
0929 00:55:57.478051 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.478054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.478101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.478162 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.479254 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.480037 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.481316 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.481664 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.481742 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.485201 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.485205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.485208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.485238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.487074 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.489882 (+ 2808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.489887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.489888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.489993 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.490325 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.490329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.490330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.490551 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.490666 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.490999 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.491139 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:55:57.495074 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3530ba2be424806b4d1767e7f4333e4. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.495214 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.496593 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3530ba2be424806b4d1767e7f4333e4. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.496737 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.498472 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.471145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.11.40.253:36723)
0929 00:55:57.471252 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.471256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.484046 (+ 12790us) server_negotiation.cc:407] Connection header received
0929 00:55:57.484230 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.484234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.484283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.484333 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.487650 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.488465 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.492154 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.492522 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.492602 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.494496 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.494500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.494503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.494531 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.496407 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.497086 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.497090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.497092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.497192 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.497492 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.497496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.497497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.497717 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.497831 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.498185 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.498311 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:55:57.504446 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.504640 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:57.504972 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.505095 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.505249 912 mvcc.cc:205] Tried to move back new op lower bound from 7334475192247152640 to 7334475192169062400. Current Snapshot: MvccSnapshot[applied={T|T < 7334475192247152640}]
I20260929 00:55:57.505236 895 mvcc.cc:205] Tried to move back new op lower bound from 7334475192247152640 to 7334475192169062400. Current Snapshot: MvccSnapshot[applied={T|T < 7334475192247152640}]
I20260929 00:55:57.505409 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:57.469981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.11.40.252:35779)
0929 00:55:57.470099 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.470103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.477688 (+ 7585us) server_negotiation.cc:407] Connection header received
0929 00:55:57.477732 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.477735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.477784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.477861 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.496989 (+ 19128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.497711 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.498825 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:57.499127 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:57.499205 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:57.501549 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:57.501553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:57.501556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:57.501585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:57.503294 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.503799 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.503803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.503805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.503889 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:57.504259 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:57.504262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:57.504264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:57.504485 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:57.504623 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:57.505133 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 00:55:57.505254 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:55:57.507539 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.507679 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:55:57.508527 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3530ba2be424806b4d1767e7f4333e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3530ba2be424806b4d1767e7f4333e4" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 33855 } } peers { permanent_uuid: "3532838e91c74538b17332ce30bb0862" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 36723 } } peers { permanent_uuid: "b9a61ac67c334e98b57711d2587f2ae8" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 35779 } } }
I20260929 00:55:57.508652 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:57.508826 913 mvcc.cc:205] Tried to move back new op lower bound from 7334475192247152640 to 7334475192169062400. Current Snapshot: MvccSnapshot[applied={T|T < 7334475192247152640 or (T in {7334475192247152640})}]
I20260929 00:55:57.519007 894 catalog_manager.cc:1347] Generated new cluster ID: cbcd554e94a34ecbb4154eda57d2c014
I20260929 00:55:57.519097 894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:57.554563 894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:57.555918 894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:57.576623 894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3530ba2be424806b4d1767e7f4333e4: Generated new TSK 0
I20260929 00:55:57.577641 894 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:55:57.610291 856 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.606926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.11.40.252:35779)
0929 00:55:57.607065 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.607070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.609192 (+ 2122us) server_negotiation.cc:407] Connection header received
0929 00:55:57.609405 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.609409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.609458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.609513 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.610147 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32950: BlockingRecv error: recv got EOF from 127.0.0.1:32950 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 00:55:57.611927 705 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.607869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.11.40.254:33855)
0929 00:55:57.607991 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.607996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.610789 (+ 2793us) server_negotiation.cc:407] Connection header received
0929 00:55:57.611167 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.611171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.611225 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.611276 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.611798 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59620: BlockingRecv error: recv got EOF from 127.0.0.1:59620 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 00:55:57.613427 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.608530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.11.40.253:36723)
0929 00:55:57.608643 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.608647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.611082 (+ 2435us) server_negotiation.cc:407] Connection header received
0929 00:55:57.612315 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.612319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.612377 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.612436 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.613291 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58296: BlockingRecv error: recv got EOF from 127.0.0.1:58296 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 00:55:57.618335 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.616625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.11.40.253:36723)
0929 00:55:57.616733 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.616737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.617291 (+ 554us) server_negotiation.cc:407] Connection header received
0929 00:55:57.617601 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.617604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.617682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.617735 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.618221 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58298: BlockingRecv error: recv got EOF from 127.0.0.1:58298 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 00:55:57.623670 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.622010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.11.40.253:36723)
0929 00:55:57.622129 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.622133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.622693 (+ 560us) server_negotiation.cc:407] Connection header received
0929 00:55:57.622902 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.622906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.622958 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.623010 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.623562 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58302: BlockingRecv error: recv got EOF from 127.0.0.1:58302 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 00:55:57.629572 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.627970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.11.40.253:36723)
0929 00:55:57.628074 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.628078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.628573 (+ 495us) server_negotiation.cc:407] Connection header received
0929 00:55:57.628780 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.628784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.628874 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.628928 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.629455 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58316: BlockingRecv error: recv got EOF from 127.0.0.1:58316 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 00:55:57.637995 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.636177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.11.40.253:36723)
0929 00:55:57.636341 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.636346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.636985 (+ 639us) server_negotiation.cc:407] Connection header received
0929 00:55:57.637190 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.637194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.637249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.637313 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.637878 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58326: BlockingRecv error: recv got EOF from 127.0.0.1:58326 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 00:55:57.650662 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.648727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.11.40.253:36723)
0929 00:55:57.648934 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.648939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.649573 (+ 634us) server_negotiation.cc:407] Connection header received
0929 00:55:57.649793 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.649797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.649856 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.649914 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.650533 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58328: BlockingRecv error: recv got EOF from 127.0.0.1:58328 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 00:55:57.671347 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.669521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.11.40.253:36723)
0929 00:55:57.669647 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.669652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.670292 (+ 640us) server_negotiation.cc:407] Connection header received
0929 00:55:57.670532 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.670536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.670591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.670647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.671210 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58342: BlockingRecv error: recv got EOF from 127.0.0.1:58342 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 00:55:57.707021 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.705912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.11.40.253:36723)
0929 00:55:57.706195 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.706199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.706215 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:55:57.706288 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.706291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.706342 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.706416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.706866 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58354: BlockingRecv error: recv got EOF from 127.0.0.1:58354 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
W20260929 00:55:57.774529 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.773214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.11.40.253:36723)
0929 00:55:57.773379 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.773386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.773596 (+ 210us) server_negotiation.cc:407] Connection header received
0929 00:55:57.773770 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.773774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.773843 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.773946 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.774404 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58370: BlockingRecv error: recv got EOF from 127.0.0.1:58370 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:55:57.906059 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:57.904757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.11.40.253:36723)
0929 00:55:57.904931 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:57.904935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:57.905091 (+ 156us) server_negotiation.cc:407] Connection header received
0929 00:55:57.905308 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:57.905311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:57.905359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:57.905459 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:57.905947 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58384: BlockingRecv error: recv got EOF from 127.0.0.1:58384 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 00:55:58.165715 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:58.164429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.11.40.253:36723)
0929 00:55:58.164585 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.164592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.164704 (+ 112us) server_negotiation.cc:407] Connection header received
0929 00:55:58.164924 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.164930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.164985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.165082 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:58.165575 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58396: BlockingRecv error: recv got EOF from 127.0.0.1:58396 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 00:55:58.469127 885 catalog_manager.cc:1259] Loaded cluster ID: cbcd554e94a34ecbb4154eda57d2c014
I20260929 00:55:58.469297 885 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: loading cluster ID for follower catalog manager: success
I20260929 00:55:58.471021 885 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: acquiring CA information for follower catalog manager: success
I20260929 00:55:58.472203 885 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3532838e91c74538b17332ce30bb0862: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:55:58.480248 910 catalog_manager.cc:1259] Loaded cluster ID: cbcd554e94a34ecbb4154eda57d2c014
I20260929 00:55:58.480329 910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: loading cluster ID for follower catalog manager: success
I20260929 00:55:58.481976 910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: acquiring CA information for follower catalog manager: success
I20260929 00:55:58.483119 910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9a61ac67c334e98b57711d2587f2ae8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:55:58.683585 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:58.679958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.11.40.253:36723)
0929 00:55:58.680204 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.680208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.680223 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:58.680352 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.680354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.680478 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.680590 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:58.680845 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.681379 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.682243 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.682457 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.682531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:58.682811 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:58.683233 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:58.683324 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:58.683390 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 00:55:58.683441 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260929 00:55:58.690217 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:58.686743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.11.40.252:35779)
0929 00:55:58.687016 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.687019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.687033 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:55:58.687131 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.687133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.687239 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.687335 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:58.687525 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.688038 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.688782 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.689066 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.689146 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:58.689410 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:58.689828 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:58.689918 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:58.689973 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 00:55:58.690019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260929 00:55:58.696956 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:58.693313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.11.40.252:35779)
0929 00:55:58.693571 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.693576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.693613 (+ 37us) server_negotiation.cc:407] Connection header received
0929 00:55:58.693652 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.693657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.693705 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.693811 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:58.694041 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.694787 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.695844 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.696059 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.696167 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:58.696471 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:58.696543 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:58.696624 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:58.696713 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 00:55:58.696762 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:55:58.697546 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:58.693386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.11.40.253:36723)
0929 00:55:58.693524 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.693530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.693764 (+ 234us) server_negotiation.cc:407] Connection header received
0929 00:55:58.693903 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.693906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.693960 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.694046 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:58.694399 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.695076 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.696202 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.696552 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.696618 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:58.697049 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:58.697134 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:58.697206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:58.697371 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 00:55:58.697427 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:55:58.698287 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:58.693008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.11.40.254:33855)
0929 00:55:58.694025 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:58.694029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:58.694041 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:55:58.694124 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:58.694127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:58.694231 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:58.694359 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:58.694591 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.695339 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.696483 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:58.696900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:58.697278 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:58.697437 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:58.697810 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:58.697924 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:58.698023 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 00:55:58.698089 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":57,"threads_started":1}
I20260929 00:55:58.702164 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 634
I20260929 00:55:58.709345 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 709
I20260929 00:55:58.715530 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 784
2026-09-29T00:55:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3153 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:58 dist-test-slave-5309 krb5kdc[1004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:58 dist-test-slave-5309 krb5kdc[1004](info): set up 2 sockets
Sep 29 00:55:58 dist-test-slave-5309 krb5kdc[1004](info): commencing operation
krb5kdc: starting...
W20260929 00:55:59.782780 11427 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/test-user.keytab.
Sep 29 00:55:59 dist-test-slave-5309 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
Entry for principal kudu/127.11.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab.
I20260929 00:55:59.915616 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34915
--webserver_interface=127.11.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:00.028272 1017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:56:00.028663 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:00.028733 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:00.028837 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:00.032903 1017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:00.033011 1017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:00.033046 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:00.033088 1017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:00.033119 1017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:00.038576 1017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.254.keytab
--principal=kudu/127.11.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.254:34915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:00.039753 1017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:00.040836 1017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:56:00.049774 1017 server_base.cc:1034] running on GCE node
W20260929 00:56:00.049796 1023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.049748 1025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.049841 1022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:00.050637 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:00.051770 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:00.052968 1017 hybrid_clock.cc:648] HybridClock initialized: now 1790643360052938 us; error 42 us; skew 500 ppm
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:00.061182 1017 init.cc:377] Logged in from keytab as kudu/127.11.40.254@KRBTEST.COM (short username kudu)
I20260929 00:56:00.062275 1017 webserver.cc:460] Webserver started at http://127.11.40.254:43839/ using document root <none> and password file <none>
I20260929 00:56:00.062876 1017 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:00.062940 1017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:00.063182 1017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:00.065395 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data/instance:
uuid: "950a99febf8b45859b3a7bbba2d3fcfd"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "a172c2c4a19e24ea7b02c3eef86da9e1"
server_key_iv: "efea7f70be55bc3bb2802c3b099b349d"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.065975 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance:
uuid: "950a99febf8b45859b3a7bbba2d3fcfd"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "a172c2c4a19e24ea7b02c3eef86da9e1"
server_key_iv: "efea7f70be55bc3bb2802c3b099b349d"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.069855 1017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:56:00.072399 1032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:00.073762 1017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:00.073911 1017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal
uuid: "950a99febf8b45859b3a7bbba2d3fcfd"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "a172c2c4a19e24ea7b02c3eef86da9e1"
server_key_iv: "efea7f70be55bc3bb2802c3b099b349d"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.074030 1017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:00.114821 1017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:00.118737 1017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:00.118978 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:00.126734 1017 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.254:34915
I20260929 00:56:00.127068 1084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.254:34915 every 8 connection(s)
I20260929 00:56:00.127763 1017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:00.130875 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:00.133405 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 1017
I20260929 00:56:00.133529 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-0/wal/instance
I20260929 00:56:00.133833 11427 external_mini_cluster.cc:1428] Setting key 8b58e8ee8bb40ec05128e9c4d24783cb
I20260929 00:56:00.135596 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.253@KRBTEST.COM" created.
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:56:00.151754 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.138306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34915 (local address 127.0.0.1:58668)
0929 00:56:00.139039 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.139092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.140069 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.140866 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.140871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.141325 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.141568 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.141582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.142775 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.142778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.143529 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.143537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.144192 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.144196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.144294 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.144936 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.144953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.146573 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.148650 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.148658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.148668 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.148967 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.149294 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.149297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.149298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.149412 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.150013 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.150019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.150161 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.150322 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.150738 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":110,"spinlock_wait_cycles":2816,"thread_start_us":169,"threads_started":1}
I20260929 00:56:00.151754 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.139369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.11.40.254:34915)
0929 00:56:00.139728 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.139747 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.139795 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:56:00.140428 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.140433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.140626 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.140854 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.141722 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.142656 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.143647 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.144030 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.144209 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.146745 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.146752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.146765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.146793 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.148510 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.149072 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.149078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.149080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.149196 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.149503 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.149506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.149508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.149899 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.150045 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.150410 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.150689 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20260929 00:56:00.154373 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:56:00.155509 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.155002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43102)
0929 00:56:00.155219 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.155351 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 00:56:00.155773 1035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
I20260929 00:56:00.156847 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.156563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43104)
0929 00:56:00.156685 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.156737 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 00:56:00.156997 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46695: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
Entry for principal kudu/127.11.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab.
I20260929 00:56:00.164558 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:46695
--webserver_interface=127.11.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:56:00.206310 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } attempt: 1
I20260929 00:56:00.207412 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.206999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43114)
0929 00:56:00.207211 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.207324 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 00:56:00.208213 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.207997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43128)
0929 00:56:00.208106 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.208156 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 00:56:00.208352 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46695: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
I20260929 00:56:00.276571 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } attempt: 2
I20260929 00:56:00.277734 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.277296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43136)
0929 00:56:00.277504 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.277589 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 00:56:00.278609 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.278359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43142)
0929 00:56:00.278477 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.278547 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 00:56:00.278796 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.253:46695: Network error: Client connection negotiation failed: client connection to 127.11.40.253:46695: connect: Connection refused (error 111)
W20260929 00:56:00.282122 1092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:56:00.282465 1092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:00.282526 1092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:00.282608 1092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:00.286804 1092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:00.286893 1092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:00.286940 1092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:00.286974 1092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:00.287006 1092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:00.292784 1092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.253.keytab
--principal=kudu/127.11.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.253:46695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:00.293982 1092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:00.295038 1092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:00.304908 1097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.304908 1098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.304909 1100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:00.305912 1092 server_base.cc:1034] running on GCE node
I20260929 00:56:00.306587 1092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:00.307898 1092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:00.309116 1092 hybrid_clock.cc:648] HybridClock initialized: now 1790643360309089 us; error 39 us; skew 500 ppm
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:00.312661 1092 init.cc:377] Logged in from keytab as kudu/127.11.40.253@KRBTEST.COM (short username kudu)
I20260929 00:56:00.313783 1092 webserver.cc:460] Webserver started at http://127.11.40.253:46015/ using document root <none> and password file <none>
I20260929 00:56:00.314409 1092 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:00.314693 1092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:00.315303 1092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:00.317938 1092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data/instance:
uuid: "9575bb1557a4458da0aa2a1f4a974c8d"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "47af0fa6e57ca0a3e0c3ae095a228ea9"
server_key_iv: "eb0d0983da657c967b576180032543c1"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.318590 1092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal/instance:
uuid: "9575bb1557a4458da0aa2a1f4a974c8d"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "47af0fa6e57ca0a3e0c3ae095a228ea9"
server_key_iv: "eb0d0983da657c967b576180032543c1"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.322820 1092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:56:00.325603 1107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:00.327014 1092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:00.327178 1092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal
uuid: "9575bb1557a4458da0aa2a1f4a974c8d"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "47af0fa6e57ca0a3e0c3ae095a228ea9"
server_key_iv: "eb0d0983da657c967b576180032543c1"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.327311 1092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:00.355055 1092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:00.358703 1092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:00.358927 1092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:00.366190 1092 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.253:46695
I20260929 00:56:00.366546 1159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.253:46695 every 8 connection(s)
I20260929 00:56:00.367275 1092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/data/info.pb
I20260929 00:56:00.369961 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } attempt: 3
I20260929 00:56:00.370410 1160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:00.371829 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 1092
I20260929 00:56:00.371948 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-1/wal/instance
I20260929 00:56:00.372268 11427 external_mini_cluster.cc:1428] Setting key 6d85258ccf568a89cae984237008a483
I20260929 00:56:00.375541 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.40.252@KRBTEST.COM; defaulting to no policy
I20260929 00:56:00.383518 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.370536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43148)
0929 00:56:00.370688 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.370707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.370847 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.373239 (+ 2392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.373242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.373459 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.373597 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.373605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.375427 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.375433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.375979 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.375988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.376707 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.376711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.376844 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.377420 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.377432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.379021 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.381454 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.381458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.381460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.381668 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.382042 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.382046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.382048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.382135 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.382932 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.382935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.383036 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.383178 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.383344 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:56:00.384613 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.371276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.11.40.253:46695)
0929 00:56:00.371709 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.371724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.371767 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:56:00.372541 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.372559 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.372968 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.373301 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.373725 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.375280 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.376104 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.376578 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.376754 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.379212 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.379217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.379226 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.379253 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.381339 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.381787 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.381798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.381804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.381912 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.382273 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.382277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.382278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.382829 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.382968 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.383344 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.383681 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.40.252@KRBTEST.COM" created.
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:56:00.386758 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:56:00.387928 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.387280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53240)
0929 00:56:00.387750 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.387837 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374}
I20260929 00:56:00.388856 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.388500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53250)
0929 00:56:00.388687 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.388744 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 00:56:00.389019 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.390714 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.377913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34915 (local address 127.0.0.1:58676)
0929 00:56:00.378438 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.378501 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.378707 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.379094 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.379103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.379607 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.379818 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.379827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.380902 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.380907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.382038 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.382102 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.382734 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.382737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.382836 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.383482 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.383501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.386570 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.388712 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.388724 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.388730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.389075 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.389430 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.389433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.389435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.389523 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.389931 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.389939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.390069 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.390213 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.390447 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260929 00:56:00.391314 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.377937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.11.40.254:34915)
0929 00:56:00.378082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.378089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.378589 (+ 500us) server_negotiation.cc:407] Connection header received
0929 00:56:00.378858 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.378861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.378918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.379013 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.379963 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.380737 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.382236 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.382576 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.382684 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.386747 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.386753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.386758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.386790 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.388544 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.389237 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.389241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.389243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.389330 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.389613 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.389617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.389619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.389822 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.389931 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.391095 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.391214 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:56:00.392326 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:56:00.402261 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.392919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43158)
0929 00:56:00.393021 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.393035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.393109 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.393337 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.393339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.393478 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.393576 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.393580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.394704 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.394707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.395246 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.395255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.395742 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.395745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.395826 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.396443 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.396458 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.398921 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.400844 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.400847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.400850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.401046 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.401320 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.401323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.401325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.401402 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.401799 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.401801 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.401892 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.401998 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.402130 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 00:56:00.402403 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.392944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.11.40.253:46695)
0929 00:56:00.393097 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.393104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.393121 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:56:00.393187 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.393191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.393241 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.393335 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.393693 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.394547 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.395357 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.395581 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.395666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.399117 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.399121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.399123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.399146 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.400717 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.401147 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.401151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.401153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.401225 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.401491 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.401495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.401496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.401698 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.401799 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.402094 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.402202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 00:56:00.403239 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
Entry for principal kudu/127.11.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab.
I20260929 00:56:00.404147 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.403758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53254)
0929 00:56:00.403966 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.404055 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 00:56:00.404753 11427 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
/tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:39817
--webserver_interface=127.11.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.40.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:00.404794 1108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.405861 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.405577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53264)
0929 00:56:00.405713 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.405777 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 00:56:00.406016 1160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.420222 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 1
I20260929 00:56:00.421203 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.420854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53274)
0929 00:56:00.421014 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.421076 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 00:56:00.422042 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.421804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53278)
0929 00:56:00.421928 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.421982 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 00:56:00.422178 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.455331 1160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 1
I20260929 00:56:00.456446 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.456067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53284)
0929 00:56:00.456296 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.456365 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 00:56:00.457230 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.457001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53298)
0929 00:56:00.457106 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.457171 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 00:56:00.457360 1160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.497362 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 2
I20260929 00:56:00.498219 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.497912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53308)
0929 00:56:00.498063 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.498137 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:56:00.499003 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.498758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53324)
0929 00:56:00.498891 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.498943 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 00:56:00.499128 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
W20260929 00:56:00.523427 1167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:56:00.523795 1167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:00.523870 1167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:00.523953 1167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:56:00.525563 1160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 2
I20260929 00:56:00.526542 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.526232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53340)
0929 00:56:00.526390 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.526464 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 00:56:00.527312 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.527080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53352)
0929 00:56:00.527194 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.527254 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:56:00.527439 1160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
W20260929 00:56:00.528865 1167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:00.528956 1167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:00.529004 1167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:00.529045 1167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:00.529098 1167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:00.535131 1167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.40.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.40.254:34915,127.11.40.253:46695,127.11.40.252:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/krb5kdc/kudu_127.11.40.252.keytab
--principal=kudu/127.11.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.40.252:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 00:56:00.536321 1167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:00.537467 1167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:00.546428 1173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.546429 1175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:00.546429 1172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:00.546756 1167 server_base.cc:1034] running on GCE node
I20260929 00:56:00.547391 1167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:00.548535 1167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:00.549731 1167 hybrid_clock.cc:648] HybridClock initialized: now 1790643360549687 us; error 55 us; skew 500 ppm
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:00.552520 1167 init.cc:377] Logged in from keytab as kudu/127.11.40.252@KRBTEST.COM (short username kudu)
I20260929 00:56:00.553545 1167 webserver.cc:460] Webserver started at http://127.11.40.252:36775/ using document root <none> and password file <none>
I20260929 00:56:00.554124 1167 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:00.554214 1167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:00.554445 1167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:56:00.556634 1167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data/instance:
uuid: "8960918df4654498ae4864a228f41c56"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "63d65a0879c909a612f0eaf994fdcb33"
server_key_iv: "503eba20baba2ef3ac5e64c0a1427a1e"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.557353 1167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal/instance:
uuid: "8960918df4654498ae4864a228f41c56"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "63d65a0879c909a612f0eaf994fdcb33"
server_key_iv: "503eba20baba2ef3ac5e64c0a1427a1e"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.561482 1167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:56:00.564253 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:00.565780 1167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:56:00.565930 1167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data,/tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal
uuid: "8960918df4654498ae4864a228f41c56"
format_stamp: "Formatted at 2026-09-29 00:56:00 on dist-test-slave-5309"
server_key: "63d65a0879c909a612f0eaf994fdcb33"
server_key_iv: "503eba20baba2ef3ac5e64c0a1427a1e"
server_key_version: "encryptionkey@0"
I20260929 00:56:00.566056 1167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:00.592088 1167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:00.595770 1167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:00.596007 1167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:00.598306 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 3
I20260929 00:56:00.599327 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.598966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53366)
0929 00:56:00.599143 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.599228 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 00:56:00.600418 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.600077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53376)
0929 00:56:00.600261 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.600338 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 00:56:00.600564 1085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.40.252:39817: Network error: Client connection negotiation failed: client connection to 127.11.40.252:39817: connect: Connection refused (error 111)
I20260929 00:56:00.604055 1167 rpc_server.cc:307] RPC server started. Bound to: 127.11.40.252:39817
I20260929 00:56:00.604403 1234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.40.252:39817 every 8 connection(s)
I20260929 00:56:00.605206 1167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/data/info.pb
I20260929 00:56:00.608225 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:00.612360 11427 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu as pid 1167
I20260929 00:56:00.612504 11427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJXuMFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790642980693194-11427-0/minicluster-data/master-2/wal/instance
I20260929 00:56:00.612843 11427 external_mini_cluster.cc:1428] Setting key 49fc702253e3238c38dac0d3bed7e119
I20260929 00:56:00.612490 1235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:56:00.618614 1160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 3
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.253@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:56:00.626899 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.614656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.11.40.254:34915)
0929 00:56:00.614853 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.614857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.615118 (+ 261us) server_negotiation.cc:407] Connection header received
0929 00:56:00.615931 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.615933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.615982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.616088 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.617095 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.617615 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.618773 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.619127 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.619226 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.622477 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.622483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.622488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.622524 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.624353 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.625105 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.625111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.625115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.625202 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.625628 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.625632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.625634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.625832 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.625938 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.626578 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.626712 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:56:00.627854 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.614523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.254:34915 (local address 127.0.0.1:58688)
0929 00:56:00.614948 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.614997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.615672 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.616132 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.616137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.616589 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.616946 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.616968 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.617726 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.617730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.618637 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.618651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.619393 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.619397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.619577 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.620456 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.620473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.622337 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.624463 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.624472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.624485 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.624952 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.625314 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.625318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.625320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.625509 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.625943 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.625952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.626271 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.626516 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.626880 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":64,"thread_start_us":167,"threads_started":1}
I20260929 00:56:00.628926 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.614317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.11.40.252:39817)
0929 00:56:00.614759 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.614780 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.614823 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:56:00.615672 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.615697 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.615954 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.616139 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.616970 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.617975 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.618917 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.619515 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.619613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.623041 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.623047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.623059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.623076 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.625521 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.625973 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.625981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.625983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.626077 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.626397 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.626400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.626402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.626742 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.628197 (+ 1455us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.628498 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.628690 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260929 00:56:00.629467 1235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:56:00.629953 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.619114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53402)
0929 00:56:00.619329 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.619345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.619433 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.619727 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.619732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.619962 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.620069 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.620075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.621291 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.621298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.622221 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.622236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.622862 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.622867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.622979 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.623438 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.623449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.624890 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.628242 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.628246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.628248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.628530 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.628873 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.628877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.628878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.628980 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.629416 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.629419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.629525 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.629635 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.629783 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 00:56:00.630577 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.619155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.11.40.252:39817)
0929 00:56:00.619451 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.619461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.619476 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:56:00.619512 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.619517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.619561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.620164 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.620235 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.621128 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.622367 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.622733 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.622854 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.625016 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.625020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.625022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.625046 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.627974 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.628665 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.628668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.628670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.628749 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.629095 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.629099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.629101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.629305 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.629404 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.630258 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.630387 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260929 00:56:00.634842 1160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: Bootstrap starting.
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.254@KRBTEST.COM
I20260929 00:56:00.638096 1160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:00.639127 1160 log.cc:824] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:00.639479 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.629985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.253:46695 (local address 127.0.0.1:43164)
0929 00:56:00.630138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.630152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.630220 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.630489 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.630492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.630706 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.630815 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.630821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.631773 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.631776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.632660 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.632670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.633373 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.633376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.633477 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.633939 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.633954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.635486 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.637599 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.637603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.637606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.637848 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.638182 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.638185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.638186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.638285 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.638855 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.638858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.638960 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.639080 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.639255 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 00:56:00.639667 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.630087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.11.40.253:46695)
0929 00:56:00.630202 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.630206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.630218 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:56:00.630326 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.630329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.630378 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.630460 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.630978 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.631652 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.632798 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.633123 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.633200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.635658 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.635664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.635668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.635699 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.637464 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.637979 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.637982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.637985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.638071 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.638383 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.638387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.638389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.638697 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.638804 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.639387 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.639520 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:56:00.640522 1235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:56:00.642303 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.631745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.11.40.254:34915)
0929 00:56:00.631882 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.631902 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.632079 (+ 177us) server_negotiation.cc:407] Connection header received
0929 00:56:00.632200 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.632203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.632248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.632344 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.633014 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.633517 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.634814 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.635073 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.635188 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.638263 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.638267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.638270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.638302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.640162 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.640763 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.640766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.640768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.640896 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.641272 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.641276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.641279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.641545 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.641790 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.641989 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.642117 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:56:00.642802 1160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: No bootstrap required, opened a new log
I20260929 00:56:00.646760 1160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.647001 1160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9575bb1557a4458da0aa2a1f4a974c8d, State: Initialized, Role: FOLLOWER
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.252@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:56:00.647342 1160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.648046 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.648307 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.649106 1160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.40.253@KRBTEST.COM
I20260929 00:56:00.651496 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.641261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.11.40.252:39817)
0929 00:56:00.641495 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.641500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.641509 (+ 9us) server_negotiation.cc:407] Connection header received
0929 00:56:00.641540 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.641553 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.641617 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.641683 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.642235 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.643010 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.644003 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.644378 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.644484 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.647521 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.647525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.647527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.647552 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.649654 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.650153 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.650169 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.650173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.650254 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.650502 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.650504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.650505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.650811 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.650914 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.651171 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.651295 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 00:56:00.651654 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.641120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53404)
0929 00:56:00.641272 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.641287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.641361 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.641787 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.641789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.642013 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.642109 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.642116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.643118 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.643121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.643864 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.643876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.644601 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.644606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.644725 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.645871 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.645887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.647383 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.649769 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.649773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.649775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.650035 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.650341 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.650344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.650345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.650423 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.650900 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.650902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.650999 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.651113 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.651495 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:56:00.654321 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.644058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.11.40.253:46695)
0929 00:56:00.644195 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.644200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.644214 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:56:00.644251 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.644253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.644295 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.644374 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.645197 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.645693 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.646790 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.647148 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.647240 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.650270 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.650276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.650279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.650308 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.652042 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.652585 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.652588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.652590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.652692 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.653039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.653043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.653044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.653255 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.653417 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.654054 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.654170 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:56:00.655016 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: Bootstrap starting.
W20260929 00:56:00.656189 1253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:56:00.656281 1253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:56:00.653849 1092 master_runner.cc:186] Error getting master registration for 127.11.40.254:34915: OK, instance_id { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" instance_seqno: 1790643360124603 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:56:00.656630 1092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:56:00.658066 1235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:00.659065 1235 log.cc:824] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:00.660032 11427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:56:00.661897 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: No bootstrap required, opened a new log
I20260929 00:56:00.661549 1059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "950a99febf8b45859b3a7bbba2d3fcfd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58692
I20260929 00:56:00.665158 1235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.665416 1235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8960918df4654498ae4864a228f41c56, State: Initialized, Role: FOLLOWER
I20260929 00:56:00.665750 1235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.666441 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.666699 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.667028 1235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:56:00.673023 1267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:56:00.673204 1267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:56:00.671079 1167 master_runner.cc:186] Error getting master registration for 127.11.40.254:34915: OK, instance_id { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" instance_seqno: 1790643360124603 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:56:00.674716 1167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:56:00.764740 1085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } attempt: 4
Sep 29 00:56:00 dist-test-slave-5309 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.40.254@KRBTEST.COM for kudu/127.11.40.252@KRBTEST.COM
I20260929 00:56:00.775576 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.765420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.40.252:39817 (local address 127.0.0.1:53414)
0929 00:56:00.765741 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:00.765757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:00.765859 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:00.766458 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:00.766461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:00.766680 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:00.766798 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.766805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.767934 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.767938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.768885 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:00.768897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.769663 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.769667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:00.769756 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.770311 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:00.770327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:00.771895 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:00.773938 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.773942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.773944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.774211 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.774502 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:00.774505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:00.774507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:00.774593 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:00.775045 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:00.775048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:00.775159 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 00:56:00.775256 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 00:56:00.775396 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260929 00:56:00.776193 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.766017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.11.40.252:39817)
0929 00:56:00.766190 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.766195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.766213 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:56:00.766264 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.766267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.766320 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.766422 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.766933 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.767792 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.769028 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.769427 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.769526 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.772053 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.772057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.772060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.772087 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.773808 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.774321 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.774325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.774326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.774408 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.774694 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.774697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.774699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.774914 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.775002 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.775903 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.776026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:56:00.779780 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: Bootstrap starting.
I20260929 00:56:00.782944 1085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:00.784101 1085 log.cc:824] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:00.786854 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: No bootstrap required, opened a new log
I20260929 00:56:00.790534 1085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.790839 1085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950a99febf8b45859b3a7bbba2d3fcfd, State: Initialized, Role: FOLLOWER
I20260929 00:56:00.791224 1085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.793176 1085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:00.793241 1059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 00:56:00.793377 1059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:00.794986 1059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.796777 1134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9575bb1557a4458da0aa2a1f4a974c8d"
I20260929 00:56:00.797076 1134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:00.795866 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.798388 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.798444 1134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950a99febf8b45859b3a7bbba2d3fcfd in term 1.
I20260929 00:56:00.799149 1059 leader_election.cc:288] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [CANDIDATE]: Term 1 election: Requested vote from peers 9575bb1557a4458da0aa2a1f4a974c8d (127.11.40.253:46695), 8960918df4654498ae4864a228f41c56 (127.11.40.252:39817)
I20260929 00:56:00.799592 1209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8960918df4654498ae4864a228f41c56"
I20260929 00:56:00.799887 1209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:00.801403 1209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950a99febf8b45859b3a7bbba2d3fcfd in term 1.
I20260929 00:56:00.801903 1033 leader_election.cc:302] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8960918df4654498ae4864a228f41c56, 950a99febf8b45859b3a7bbba2d3fcfd; no voters:
W20260929 00:56:00.803659 1281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:56:00.803820 1281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:56:00.804581 1270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:00.805724 1270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [term 1 LEADER]: Becoming Leader. State: Replica: 950a99febf8b45859b3a7bbba2d3fcfd, State: Running, Role: LEADER
I20260929 00:56:00.806118 1270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } }
I20260929 00:56:00.808068 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950a99febf8b45859b3a7bbba2d3fcfd. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.808172 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:00.808696 1017 master_runner.cc:263] No leader master found from master 950a99febf8b45859b3a7bbba2d3fcfd
I20260929 00:56:00.808763 1017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:56:00.810729 1289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:00.811633 1289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:00.819779 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.808731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.11.40.253:46695)
0929 00:56:00.808880 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.808884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.809412 (+ 528us) server_negotiation.cc:407] Connection header received
0929 00:56:00.809615 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.809618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.809669 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.809724 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.810885 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.811769 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.813101 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.813460 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.813538 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.815776 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.815780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.815783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.815815 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.817745 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.818296 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.818301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.818304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.818411 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.818735 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.818739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.818740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.818969 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.819085 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.819476 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.819607 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:56:00.828465 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.809483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.11.40.254:34915)
0929 00:56:00.809583 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.809587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.814496 (+ 4909us) server_negotiation.cc:407] Connection header received
0929 00:56:00.814663 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.814666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.814714 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.814763 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.817933 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.818678 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.821905 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.822231 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.822318 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.824286 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.824291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.824294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.824328 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.826396 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.826916 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.826920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.826926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.827018 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.827362 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.827366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.827368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.827598 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.827730 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.828142 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.828278 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 00:56:00.835561 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:00.810130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.11.40.252:39817)
0929 00:56:00.810233 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.810237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.816760 (+ 6523us) server_negotiation.cc:407] Connection header received
0929 00:56:00.816968 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.816972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.817039 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.817092 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.819872 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.820635 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.828925 (+ 8290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:00.829302 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:00.829382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:00.831504 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:00.831509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:00.831512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:00.831545 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:00.833421 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.833973 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.833977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.833979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.834075 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:00.834438 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:00.834442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:00.834444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:00.834691 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:00.834811 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:00.835252 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 00:56:00.835385 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:56:00.835634 1209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [term 1 FOLLOWER]: Refusing update from remote peer 950a99febf8b45859b3a7bbba2d3fcfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:00.836411 1287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:00.841756 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950a99febf8b45859b3a7bbba2d3fcfd. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.841912 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.843606 1134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [term 1 FOLLOWER]: Refusing update from remote peer 950a99febf8b45859b3a7bbba2d3fcfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:00.844343 1287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:56:00.847568 1188 tablet.cc:2369] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:00.849261 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950a99febf8b45859b3a7bbba2d3fcfd. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.849380 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.851488 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.851639 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:00.851907 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.852011 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.851984 1293 mvcc.cc:205] Tried to move back new op lower bound from 7334475205973909504 to 7334475205865705472. Current Snapshot: MvccSnapshot[applied={T|T < 7334475205973909504}]
I20260929 00:56:00.854029 1287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.854192 1287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:00.854233 1289 catalog_manager.cc:1347] Generated new cluster ID: b6fa178da35245fda4123188d7f5cfd4
I20260929 00:56:00.854274 1289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:00.855479 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950a99febf8b45859b3a7bbba2d3fcfd" member_type: VOTER last_known_addr { host: "127.11.40.254" port: 34915 } } peers { permanent_uuid: "9575bb1557a4458da0aa2a1f4a974c8d" member_type: VOTER last_known_addr { host: "127.11.40.253" port: 46695 } } peers { permanent_uuid: "8960918df4654498ae4864a228f41c56" member_type: VOTER last_known_addr { host: "127.11.40.252" port: 39817 } } }
I20260929 00:56:00.855602 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:56:00.856199 1294 mvcc.cc:205] Tried to move back new op lower bound from 7334475205973909504 to 7334475205865705472. Current Snapshot: MvccSnapshot[applied={T|T < 7334475205973909504}]
I20260929 00:56:00.865751 1289 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:00.866739 1289 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:00.879410 1289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950a99febf8b45859b3a7bbba2d3fcfd: Generated new TSK 0
I20260929 00:56:00.880367 1289 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:56:00.904247 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.902405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.11.40.253:46695)
0929 00:56:00.902537 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.902541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.903084 (+ 543us) server_negotiation.cc:407] Connection header received
0929 00:56:00.903315 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.903318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.903376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.903431 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.904115 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43186: BlockingRecv error: recv got EOF from 127.0.0.1:43186 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 00:56:00.904248 1089 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.902756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.11.40.254:34915)
0929 00:56:00.902911 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.902916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.903082 (+ 166us) server_negotiation.cc:407] Connection header received
0929 00:56:00.903322 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.903326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.903387 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.903448 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.904114 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58718: BlockingRecv error: recv got EOF from 127.0.0.1:58718 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:56:00.904248 1240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.902528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.11.40.252:39817)
0929 00:56:00.902663 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.902668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.903248 (+ 580us) server_negotiation.cc:407] Connection header received
0929 00:56:00.903440 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.903443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.903493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.903546 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.904103 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53426: BlockingRecv error: recv got EOF from 127.0.0.1:53426 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 00:56:00.907759 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.906687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.11.40.253:46695)
0929 00:56:00.906807 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.906810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.906862 (+ 52us) server_negotiation.cc:407] Connection header received
0929 00:56:00.907048 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.907050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.907115 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.907187 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.907623 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43202: BlockingRecv error: recv got EOF from 127.0.0.1:43202 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 00:56:00.911724 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.910870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.11.40.253:46695)
0929 00:56:00.910986 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.910993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.911004 (+ 11us) server_negotiation.cc:407] Connection header received
0929 00:56:00.911047 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.911049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.911101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.911191 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.911599 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43206: BlockingRecv error: recv got EOF from 127.0.0.1:43206 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 00:56:00.916581 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.915496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.11.40.253:46695)
0929 00:56:00.915621 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.915625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.915768 (+ 143us) server_negotiation.cc:407] Connection header received
0929 00:56:00.915918 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.915921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.915972 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.916059 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.916475 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 00:56:00.923681 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.922503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.11.40.253:46695)
0929 00:56:00.922696 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.922700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.922896 (+ 196us) server_negotiation.cc:407] Connection header received
0929 00:56:00.923013 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.923016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.923078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.923182 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.923563 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43224: BlockingRecv error: recv got EOF from 127.0.0.1:43224 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 00:56:00.935272 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.933960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.11.40.253:46695)
0929 00:56:00.934129 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.934135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.934295 (+ 160us) server_negotiation.cc:407] Connection header received
0929 00:56:00.934431 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.934434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.934482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.934558 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.935123 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43234: BlockingRecv error: recv got EOF from 127.0.0.1:43234 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 00:56:00.954978 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.953727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.11.40.253:46695)
0929 00:56:00.953894 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.953899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.954115 (+ 216us) server_negotiation.cc:407] Connection header received
0929 00:56:00.954232 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.954236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.954303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.954422 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.954852 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43236: BlockingRecv error: recv got EOF from 127.0.0.1:43236 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 00:56:00.990761 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:00.989396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.11.40.253:46695)
0929 00:56:00.989545 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:00.989549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:00.989703 (+ 154us) server_negotiation.cc:407] Connection header received
0929 00:56:00.989854 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:00.989856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:00.989908 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:00.989985 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:00.990607 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43248: BlockingRecv error: recv got EOF from 127.0.0.1:43248 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:56:01.058579 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:01.057077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.11.40.253:46695)
0929 00:56:01.057316 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.057322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.057497 (+ 175us) server_negotiation.cc:407] Connection header received
0929 00:56:01.057740 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.057744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.057807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.057913 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:01.058445 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43264: BlockingRecv error: recv got EOF from 127.0.0.1:43264 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260929 00:56:01.190683 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:01.189362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.11.40.253:46695)
0929 00:56:01.189502 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.189507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.189713 (+ 206us) server_negotiation.cc:407] Connection header received
0929 00:56:01.189901 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.189904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.189957 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.190052 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:01.190530 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43278: BlockingRecv error: recv got EOF from 127.0.0.1:43278 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 00:56:01.450729 1162 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:56:01.449311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.11.40.253:46695)
0929 00:56:01.449503 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.449509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.449613 (+ 104us) server_negotiation.cc:407] Connection header received
0929 00:56:01.449798 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.449802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.449858 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.449949 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:01.450549 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43290: BlockingRecv error: recv got EOF from 127.0.0.1:43290 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:56:01.657711 1253 catalog_manager.cc:1259] Loaded cluster ID: b6fa178da35245fda4123188d7f5cfd4
I20260929 00:56:01.657809 1253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: loading cluster ID for follower catalog manager: success
I20260929 00:56:01.659590 1253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: acquiring CA information for follower catalog manager: success
I20260929 00:56:01.660938 1253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9575bb1557a4458da0aa2a1f4a974c8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:56:01.674628 1267 catalog_manager.cc:1259] Loaded cluster ID: b6fa178da35245fda4123188d7f5cfd4
I20260929 00:56:01.674715 1267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: loading cluster ID for follower catalog manager: success
I20260929 00:56:01.676513 1267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: acquiring CA information for follower catalog manager: success
I20260929 00:56:01.678349 1267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8960918df4654498ae4864a228f41c56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:56:01.970086 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:01.965491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.11.40.253:46695)
0929 00:56:01.965811 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.965816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.965938 (+ 122us) server_negotiation.cc:407] Connection header received
0929 00:56:01.966130 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.966133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.966271 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.966393 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:01.966674 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.967324 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.968304 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.968554 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.968680 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:01.969096 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:01.969616 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:01.969774 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:01.969854 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 00:56:01.969916 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260929 00:56:01.978245 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:01.973723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.11.40.252:39817)
0929 00:56:01.973975 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.973980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.973992 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:56:01.974079 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.974081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.974199 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.974303 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:01.974683 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.975249 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.976360 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.976596 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.976690 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:01.977138 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:01.977716 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:01.977904 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:01.977972 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 00:56:01.978041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
I20260929 00:56:01.985715 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:01.982039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.11.40.252:39817)
0929 00:56:01.982158 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.982162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.982239 (+ 77us) server_negotiation.cc:407] Connection header received
0929 00:56:01.982416 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.982421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.982480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.982563 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:01.982772 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.983528 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.984525 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.984749 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.984898 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:01.985201 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:01.985284 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:01.985327 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:01.985492 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 00:56:01.985560 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 00:56:01.986589 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:01.981835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.11.40.253:46695)
0929 00:56:01.982205 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.982210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.983152 (+ 942us) server_negotiation.cc:407] Connection header received
0929 00:56:01.983270 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.983274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.983347 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.983430 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:01.983759 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.984467 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.985490 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.985728 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.985926 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:01.986102 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:01.986181 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:01.986276 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:01.986399 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:56:01.986464 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260929 00:56:01.986828 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:01.981824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.11.40.254:34915)
0929 00:56:01.982392 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:01.982397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:01.982409 (+ 12us) server_negotiation.cc:407] Connection header received
0929 00:56:01.982507 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:01.982512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:01.982694 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:01.982835 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:01.983027 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.983926 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.985126 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:01.985485 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:01.985575 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:01.986010 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:01.986416 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:01.986523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:01.986606 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 00:56:01.986667 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":79,"threads_started":1}
I20260929 00:56:01.990434 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 1017
I20260929 00:56:01.998044 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 1092
I20260929 00:56:02.004572 11427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJXuMFZ/build/debug/bin/kudu with pid 1167
2026-09-29T00:56:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3290 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6443 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381294 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS